Nov 02 22:49:46.760376 ip-10-0-139-234 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 02 22:49:46.760386 ip-10-0-139-234 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 02 22:49:46.760393 ip-10-0-139-234 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 02 22:49:46.760620 ip-10-0-139-234 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 02 22:49:57.009690 ip-10-0-139-234 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 02 22:49:57.009711 ip-10-0-139-234 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 1b9479a74fc04a45a5b465f211d57cdd --
Nov 02 22:51:59.267665 ip-10-0-139-234 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 22:51:59.621825 ip-10-0-139-234 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 22:51:59.621825 ip-10-0-139-234 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 22:51:59.621825 ip-10-0-139-234 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 22:51:59.621825 ip-10-0-139-234 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 22:51:59.625457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625375    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 22:51:59.625457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625457    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 22:51:59.625527 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625462    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 22:51:59.625527 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625465    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 22:51:59.625527 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625468    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 22:51:59.625527 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625471    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 22:51:59.625527 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625474    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 22:51:59.625527 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625477    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 22:51:59.625527 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625479    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 22:51:59.625527 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625482    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 22:51:59.625527 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625484    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 22:51:59.625527 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625487    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 22:51:59.625527 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625489    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 22:51:59.625527 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625492    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 22:51:59.625527 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625495    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 22:51:59.625527 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625497    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 22:51:59.625527 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625500    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 22:51:59.625527 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625502    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 22:51:59.625527 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625505    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 22:51:59.625527 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625509    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 22:51:59.625527 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625512    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 22:51:59.625527 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625514    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 22:51:59.626008 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625517    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 22:51:59.626008 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625520    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 22:51:59.626008 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625522    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 22:51:59.626008 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625525    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 22:51:59.626008 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625527    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 22:51:59.626008 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625530    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 22:51:59.626008 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625534    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 22:51:59.626008 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625537    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 22:51:59.626008 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625540    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 22:51:59.626008 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625542    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 22:51:59.626008 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625545    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 22:51:59.626008 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625548    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 22:51:59.626008 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625551    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 22:51:59.626008 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625553    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 22:51:59.626008 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625556    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 22:51:59.626008 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625558    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 22:51:59.626008 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625561    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 22:51:59.626008 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625564    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 22:51:59.626008 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625567    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 22:51:59.626457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625570    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 22:51:59.626457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625572    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 22:51:59.626457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625575    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 22:51:59.626457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625578    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 22:51:59.626457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625580    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 22:51:59.626457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625582    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 22:51:59.626457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625585    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 22:51:59.626457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625587    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 22:51:59.626457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625590    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 22:51:59.626457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625592    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 22:51:59.626457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625595    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 22:51:59.626457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625597    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 22:51:59.626457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625600    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 22:51:59.626457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625602    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 22:51:59.626457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625606    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 22:51:59.626457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625609    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 22:51:59.626457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625611    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 22:51:59.626457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625614    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 22:51:59.626457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625617    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 22:51:59.626457 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625619    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 22:51:59.626947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625621    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 22:51:59.626947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625624    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 22:51:59.626947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625627    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 22:51:59.626947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625630    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 22:51:59.626947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625632    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 22:51:59.626947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625634    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 22:51:59.626947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625637    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 22:51:59.626947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625639    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 22:51:59.626947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625642    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 22:51:59.626947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625644    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 22:51:59.626947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625647    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 22:51:59.626947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625656    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 22:51:59.626947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625661    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 22:51:59.626947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625666    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 22:51:59.626947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625669    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 22:51:59.626947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625672    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 22:51:59.626947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625674    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 22:51:59.626947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625677    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 22:51:59.626947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625680    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 22:51:59.627425 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625682    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 22:51:59.627425 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625685    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 22:51:59.627425 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625687    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 22:51:59.627425 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625690    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 22:51:59.627425 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625692    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 22:51:59.627425 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625695    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 22:51:59.627425 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625698    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 22:51:59.627425 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625702    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 22:51:59.627425 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625705    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 22:51:59.627425 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625708    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 22:51:59.627425 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625711    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 22:51:59.627425 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625713    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 22:51:59.627425 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625716    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 22:51:59.627425 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625718    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 22:51:59.627425 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625721    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 22:51:59.627425 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625724    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 22:51:59.627425 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625726    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 22:51:59.627425 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625728    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 22:51:59.627425 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625731    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 22:51:59.627425 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625733    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 22:51:59.627913 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625736    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 22:51:59.627913 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625738    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 22:51:59.627913 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625741    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 22:51:59.627913 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625744    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 22:51:59.627913 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625749    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 22:51:59.627913 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.625759    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 22:51:59.627913 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626567    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 22:51:59.627913 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626574    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 22:51:59.627913 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626578    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 22:51:59.627913 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626580    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 22:51:59.627913 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626584    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 22:51:59.627913 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626587    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 22:51:59.627913 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626589    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 22:51:59.627913 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626592    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 22:51:59.627913 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626594    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 22:51:59.627913 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626597    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 22:51:59.627913 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626600    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 22:51:59.627913 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626603    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 22:51:59.627913 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626605    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 22:51:59.627913 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626609    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 22:51:59.628407 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626611    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 22:51:59.628407 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626614    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 22:51:59.628407 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626617    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 22:51:59.628407 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626619    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 22:51:59.628407 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626622    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 22:51:59.628407 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626624    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 22:51:59.628407 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626627    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 22:51:59.628407 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626630    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 22:51:59.628407 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626633    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 22:51:59.628407 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626635    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 22:51:59.628407 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626637    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 22:51:59.628407 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626640    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 22:51:59.628407 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626643    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 22:51:59.628407 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626645    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 22:51:59.628407 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626648    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 22:51:59.628407 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626650    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 22:51:59.628407 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626653    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 22:51:59.628407 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626655    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 22:51:59.628407 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626665    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 22:51:59.628407 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626668    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 22:51:59.628887 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626670    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 22:51:59.628887 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626673    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 22:51:59.628887 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626676    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 22:51:59.628887 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626678    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 22:51:59.628887 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626681    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 22:51:59.628887 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626684    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 22:51:59.628887 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626686    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 22:51:59.628887 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626690    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 22:51:59.628887 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626694    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 22:51:59.628887 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626697    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 22:51:59.628887 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626700    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 22:51:59.628887 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626703    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 22:51:59.628887 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626706    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 22:51:59.628887 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626709    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 22:51:59.628887 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626712    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 22:51:59.628887 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626714    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 22:51:59.628887 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626717    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 22:51:59.628887 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626720    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 22:51:59.628887 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626723    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 22:51:59.629350 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626726    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 22:51:59.629350 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626729    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 22:51:59.629350 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626733    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 22:51:59.629350 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626737    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 22:51:59.629350 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626741    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 22:51:59.629350 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626743    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 22:51:59.629350 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626746    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 22:51:59.629350 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626748    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 22:51:59.629350 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626751    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 22:51:59.629350 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626753    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 22:51:59.629350 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626756    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 22:51:59.629350 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626758    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 22:51:59.629350 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626761    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 22:51:59.629350 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626764    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 22:51:59.629350 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626766    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 22:51:59.629350 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626769    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 22:51:59.629350 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626772    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 22:51:59.629350 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626775    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 22:51:59.629350 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626777    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 22:51:59.629837 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626780    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 22:51:59.629837 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626783    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 22:51:59.629837 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626785    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 22:51:59.629837 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626788    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 22:51:59.629837 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626790    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 22:51:59.629837 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626793    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 22:51:59.629837 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626796    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 22:51:59.629837 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626799    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 22:51:59.629837 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626801    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 22:51:59.629837 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626804    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 22:51:59.629837 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626806    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 22:51:59.629837 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626809    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 22:51:59.629837 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626812    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 22:51:59.629837 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626814    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 22:51:59.629837 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626817    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 22:51:59.629837 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626820    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 22:51:59.629837 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626822    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 22:51:59.629837 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626825    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 22:51:59.629837 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626827    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 22:51:59.629837 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626830    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 22:51:59.630316 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626832    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 22:51:59.630316 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626835    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 22:51:59.630316 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626838    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 22:51:59.630316 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626840    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 22:51:59.630316 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626844    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 22:51:59.630316 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626846    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 22:51:59.630316 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626849    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 22:51:59.630316 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626852    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 22:51:59.630316 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626854    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 22:51:59.630316 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626857    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 22:51:59.630316 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626860    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 22:51:59.630316 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626862    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 22:51:59.630316 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626865    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 22:51:59.630316 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.626867    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 22:51:59.630316 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627385    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 22:51:59.630316 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627392    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 22:51:59.630316 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627395    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 22:51:59.630316 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627398    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 22:51:59.630316 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627401    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 22:51:59.630778 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627404    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 22:51:59.630778 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627407    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 22:51:59.630778 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627410    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 22:51:59.630778 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627413    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 22:51:59.630778 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627416    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 22:51:59.630778 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627419    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 22:51:59.630778 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627421    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 22:51:59.630778 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627425    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 22:51:59.630778 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627428    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 22:51:59.630778 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627434    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 22:51:59.630778 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627437    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 22:51:59.630778 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627439    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 22:51:59.630778 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627442    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 22:51:59.630778 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627445    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 22:51:59.630778 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627448    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 22:51:59.630778 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627451    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 22:51:59.630778 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627453    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 22:51:59.630778 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627456    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 22:51:59.630778 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627459    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 22:51:59.630778 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627462    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 22:51:59.631244 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627465    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 22:51:59.631244 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627467    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 22:51:59.631244 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627470    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 22:51:59.631244 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627472    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 22:51:59.631244 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627476    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 22:51:59.631244 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627480    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 22:51:59.631244 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627483    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 22:51:59.631244 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627487    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 22:51:59.631244 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627489    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 22:51:59.631244 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627494    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 22:51:59.631244 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627496    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 22:51:59.631244 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627499    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 22:51:59.631244 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627502    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 22:51:59.631244 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627504    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 22:51:59.631244 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627507    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 22:51:59.631244 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627509    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 22:51:59.631244 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627512    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 22:51:59.631244 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627514    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 22:51:59.631698 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627517    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 22:51:59.631698 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627519    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 22:51:59.631698 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627522    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 22:51:59.631698 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627526    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 22:51:59.631698 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627528    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 22:51:59.631698 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627531    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 22:51:59.631698 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627533    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 22:51:59.631698 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627536    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 22:51:59.631698 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627539    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 22:51:59.631698 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627541    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 22:51:59.631698 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627544    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 22:51:59.631698 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627546    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 22:51:59.631698 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627549    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 22:51:59.631698 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627551    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 22:51:59.631698 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627554    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 22:51:59.631698 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627557    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 22:51:59.631698 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627559    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 22:51:59.631698 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627562    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 22:51:59.631698 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627564    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 22:51:59.631698 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627567    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 22:51:59.632163 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627570    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 22:51:59.632163 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627572    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 22:51:59.632163 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627574    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 22:51:59.632163 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627578    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 22:51:59.632163 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627581    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 22:51:59.632163 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627583    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 22:51:59.632163 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627586    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 22:51:59.632163 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627588    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 22:51:59.632163 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627591    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 22:51:59.632163 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627593    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 22:51:59.632163 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627596    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 22:51:59.632163 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627598    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 22:51:59.632163 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627601    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 22:51:59.632163 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627603    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 22:51:59.632163 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627606    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 22:51:59.632163 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627612    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 22:51:59.632163 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627615    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 22:51:59.632163 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627618    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 22:51:59.632163 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627621    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 22:51:59.632642 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627624    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 22:51:59.632642 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627626    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 22:51:59.632642 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627629    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 22:51:59.632642 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627631    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 22:51:59.632642 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627634    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 22:51:59.632642 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627636    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 22:51:59.632642 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627639    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 22:51:59.632642 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627641    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 22:51:59.632642 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627644    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 22:51:59.632642 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627647    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 22:51:59.632642 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627649    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 22:51:59.632642 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627652    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 22:51:59.632642 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627654    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 22:51:59.632642 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627657    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 22:51:59.632642 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627659    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 22:51:59.632642 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627662    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 22:51:59.632642 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627665    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 22:51:59.632642 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627676    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 22:51:59.632642 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627679    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 22:51:59.632642 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627681    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627684    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627686    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627689    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.627691    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628910    2582 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628919    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628925    2582 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628930    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628936    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628939    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628943    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628948    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628951    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628954    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628958    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628961    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628964    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628967    2582 flags.go:64] FLAG: --cgroup-root=""
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628970    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628973    2582 flags.go:64] FLAG: --client-ca-file=""
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628976    2582 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628979    2582 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628984    2582 flags.go:64] FLAG: --cluster-domain=""
Nov 02 22:51:59.633154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628988    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628991    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628995    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.628998    2582 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629002    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629006    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629009    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629012    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629015    2582 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629019    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629022    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629025    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629028    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629032    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629036    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629038    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629041    2582 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629044    2582 flags.go:64] FLAG: --enable-server="true"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629047    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629051    2582 flags.go:64] FLAG: --event-burst="100"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629055    2582 flags.go:64] FLAG: --event-qps="50"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629057    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629061    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629064    2582 flags.go:64] FLAG: --eviction-hard=""
Nov 02 22:51:59.633738 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629068    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629071    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629074    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629077    2582 flags.go:64] FLAG: --eviction-soft=""
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629081    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629084    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629087    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629090    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629094    2582 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629097    2582 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629100    2582 flags.go:64] FLAG: --feature-gates=""
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629104    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629107    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629110    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629113    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629117    2582 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629120    2582 flags.go:64] FLAG: --help="false"
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629123    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629126    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629129    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629132    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629136    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629139    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 22:51:59.634337 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629142    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629145    2582 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629148    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629151    2582 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629154    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629157    2582 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629160    2582 flags.go:64] FLAG: --kube-reserved=""
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629163    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629166    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629169    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629172    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629174    2582 flags.go:64] FLAG: --lock-file=""
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629177    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629180    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629183    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629189    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629192    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629196    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629199    2582 flags.go:64] FLAG: --logging-format="text"
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629202    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629205    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629208    2582 flags.go:64] FLAG: --manifest-url=""
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629211    2582 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629215    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629218    2582 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 22:51:59.634905 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629222    2582 flags.go:64] FLAG: --max-pods="110"
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629225    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629228    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629231    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629235    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629238    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629240    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629244    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629250    2582 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629253    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629256    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629259    2582 flags.go:64] FLAG: --pod-cidr=""
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629262    2582 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629265    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629268    2582 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629271    2582 flags.go:64] FLAG: --port="10250"
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629274    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629277    2582 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0e400f65c79abf5da"
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629280    2582 flags.go:64] FLAG: --qos-reserved=""
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629283    2582 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629286    2582 flags.go:64] FLAG: --register-node="true"
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629289    2582 flags.go:64] FLAG: --register-with-taints=""
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629293    2582 flags.go:64] FLAG: --registry-burst="10"
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629296    2582 flags.go:64] FLAG: --registry-qps="5"
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629299    2582 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629302    2582 flags.go:64] FLAG: --reserved-memory=""
Nov 02 22:51:59.635551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629306    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629309    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629312    2582 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629315    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629318    2582 flags.go:64] FLAG: --runonce="false"
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629321    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629336    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629340    2582 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629343    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629346    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629349    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629352    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629356    2582 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629359    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629362    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629365    2582 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629368    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629371    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629374    2582 flags.go:64] FLAG: --system-cgroups=""
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629377    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629382    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629385    2582 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629388    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629393    2582 flags.go:64] FLAG: --tls-min-version=""
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629396    2582 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 22:51:59.636168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629399    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629401    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629406    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629409    2582 flags.go:64] FLAG: --v="2"
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629413    2582 flags.go:64] FLAG: --version="false"
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629418    2582 flags.go:64] FLAG: --vmodule=""
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629422    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629425    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629559    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629562    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629565    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629568    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629570    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629574    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629577    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629579    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629582    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629585    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629587    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629590    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629592    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 22:51:59.636755 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629595    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 22:51:59.637246 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629598    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 22:51:59.637246 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629600    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 22:51:59.637246 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629603    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 22:51:59.637246 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629606    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 22:51:59.637246 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629609    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 22:51:59.637246 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629611    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 22:51:59.637246 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629614    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 22:51:59.637246 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629616    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 22:51:59.637246 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629619    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 22:51:59.637246 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629622    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 22:51:59.637246 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629625    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 22:51:59.637246 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629627    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 22:51:59.637246 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629631    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 22:51:59.637246 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629634    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 22:51:59.637246 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629637    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 22:51:59.637246 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629639    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 22:51:59.637246 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629642    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 22:51:59.637246 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629644    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 22:51:59.637246 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629648    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 22:51:59.637246 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629650    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 22:51:59.637861 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629653    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 22:51:59.637861 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629657    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 22:51:59.637861 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629661    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 22:51:59.637861 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629666    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 22:51:59.637861 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629668    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 22:51:59.637861 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629671    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 22:51:59.637861 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629675    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 22:51:59.637861 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629678    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 22:51:59.637861 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629680    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 22:51:59.637861 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629683    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 22:51:59.637861 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629686    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 22:51:59.637861 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629688    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 22:51:59.637861 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629691    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 22:51:59.637861 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629694    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 22:51:59.637861 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629696    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 22:51:59.637861 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629699    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 22:51:59.637861 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629701    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 22:51:59.637861 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629704    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 22:51:59.637861 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629707    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 22:51:59.638585 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629709    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 22:51:59.638585 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629712    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 22:51:59.638585 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629715    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 22:51:59.638585 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629718    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 22:51:59.638585 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629720    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 22:51:59.638585 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629724    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 22:51:59.638585 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629727    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 22:51:59.638585 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629729    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 22:51:59.638585 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629732    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 22:51:59.638585 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629734    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 22:51:59.638585 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629737    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 22:51:59.638585 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629739    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 22:51:59.638585 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629742    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 22:51:59.638585 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629745    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 22:51:59.638585 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629747    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 22:51:59.638585 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629750    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 22:51:59.638585 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629754    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 22:51:59.638585 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629756    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 22:51:59.638585 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629759    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 22:51:59.638585 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629761    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 22:51:59.639257 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629764    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 22:51:59.639257 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629767    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 22:51:59.639257 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629769    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 22:51:59.639257 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629772    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 22:51:59.639257 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629775    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 22:51:59.639257 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629777    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 22:51:59.639257 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629780    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 22:51:59.639257 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629782    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 22:51:59.639257 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629785    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 22:51:59.639257 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629789    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 22:51:59.639257 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629792    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 22:51:59.639257 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629795    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 22:51:59.639257 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629798    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 22:51:59.639257 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629801    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 22:51:59.639257 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629803    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 22:51:59.639257 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629806    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 22:51:59.639257 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629808    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 22:51:59.639257 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629812    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 22:51:59.639257 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629815    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 22:51:59.639757 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629818    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 22:51:59.639757 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629820    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 22:51:59.639757 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629823    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 22:51:59.639757 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629825    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 22:51:59.639757 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629828    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 22:51:59.639757 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629830    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 22:51:59.639757 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629833    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 22:51:59.639757 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629836    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 22:51:59.639757 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629838    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 22:51:59.639757 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629842    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 22:51:59.639757 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629845    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 22:51:59.639757 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629848    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 22:51:59.639757 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629851    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 22:51:59.639757 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629853    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 22:51:59.639757 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.629878    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 22:51:59.640125 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629914    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 22:51:59.640125 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629918    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 22:51:59.640125 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629922    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 22:51:59.640125 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629925    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 22:51:59.640125 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629928    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 22:51:59.640125 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629933    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 22:51:59.640125 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629936    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 22:51:59.640125 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629939    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 22:51:59.640125 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629942    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 22:51:59.640125 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629945    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 22:51:59.640125 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629947    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 22:51:59.640125 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629950    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 22:51:59.640125 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629953    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 22:51:59.640125 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629955    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 22:51:59.640125 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629958    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 22:51:59.640125 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629963    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 22:51:59.640125 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629965    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 22:51:59.640125 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629968    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 22:51:59.640125 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629971    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 22:51:59.640618 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629973    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 22:51:59.640618 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629976    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 22:51:59.640618 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629979    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 22:51:59.640618 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629981    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 22:51:59.640618 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629984    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 22:51:59.640618 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629987    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 22:51:59.640618 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629990    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 22:51:59.640618 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629993    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 22:51:59.640618 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.629997    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 22:51:59.640618 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630000    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 22:51:59.640618 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630002    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 22:51:59.640618 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630005    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 22:51:59.640618 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630008    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 22:51:59.640618 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630010    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 22:51:59.640618 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630013    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 22:51:59.640618 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630015    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 22:51:59.640618 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630018    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 22:51:59.640618 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630021    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 22:51:59.640618 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630023    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 22:51:59.640618 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630026    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 22:51:59.641087 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630028    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 22:51:59.641087 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630031    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 22:51:59.641087 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630033    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 22:51:59.641087 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630036    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 22:51:59.641087 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630039    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 22:51:59.641087 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630041    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 22:51:59.641087 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630044    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 22:51:59.641087 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630046    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 22:51:59.641087 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630051    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 22:51:59.641087 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630053    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 22:51:59.641087 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630056    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 22:51:59.641087 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630058    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 22:51:59.641087 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630061    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 22:51:59.641087 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630063    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 22:51:59.641087 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630066    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 22:51:59.641087 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630069    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 22:51:59.641087 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630071    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 22:51:59.641087 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630073    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 22:51:59.641087 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630077    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 22:51:59.641087 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630079    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 22:51:59.641573 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630083    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 22:51:59.641573 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630086    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 22:51:59.641573 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630089    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 22:51:59.641573 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630091    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 22:51:59.641573 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630094    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 22:51:59.641573 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630099    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 22:51:59.641573 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630103    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 22:51:59.641573 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630106    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 22:51:59.641573 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630109    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 22:51:59.641573 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630112    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 22:51:59.641573 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630115    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 22:51:59.641573 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630118    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 22:51:59.641573 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630120    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 22:51:59.641573 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630123    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 22:51:59.641573 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630126    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 22:51:59.641573 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630128    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 22:51:59.641573 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630131    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 22:51:59.641573 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630134    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 22:51:59.641573 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630153    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 22:51:59.642031 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630157    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 22:51:59.642031 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630162    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 22:51:59.642031 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630166    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 22:51:59.642031 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630168    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 22:51:59.642031 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630171    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 22:51:59.642031 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630174    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 22:51:59.642031 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630177    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 22:51:59.642031 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630180    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 22:51:59.642031 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630182    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 22:51:59.642031 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630185    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 22:51:59.642031 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630187    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 22:51:59.642031 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630190    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 22:51:59.642031 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630193    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 22:51:59.642031 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630197    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 22:51:59.642031 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630199    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 22:51:59.642031 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630202    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 22:51:59.642031 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630204    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 22:51:59.642031 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630207    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 22:51:59.642031 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630209    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 22:51:59.642031 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630212    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 22:51:59.642543 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630214    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 22:51:59.642543 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630217    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 22:51:59.642543 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630220    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 22:51:59.642543 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630222    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 22:51:59.642543 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630225    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 22:51:59.642543 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630227    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 22:51:59.642543 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630230    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 22:51:59.642543 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.630232    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 22:51:59.642543 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.637056    2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 22:51:59.642543 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.637073    2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 22:51:59.642543 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.637084    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 22:51:59.642543 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.637089    2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 22:51:59.642543 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637157    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 22:51:59.642543 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637161    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 22:51:59.642543 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637164    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 22:51:59.642543 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637167    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 22:51:59.642543 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637170    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 22:51:59.642543 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637173    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 22:51:59.642543 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637175    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 22:51:59.643003 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637178    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 22:51:59.643003 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637181    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 22:51:59.643003 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637183    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 22:51:59.643003 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637186    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 22:51:59.643003 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637189    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 22:51:59.643003 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637192    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 22:51:59.643003 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637195    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 22:51:59.643003 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637197    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 22:51:59.643003 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637200    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 22:51:59.643003 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637203    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 22:51:59.643003 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637205    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 22:51:59.643003 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637208    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 22:51:59.643003 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637210    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 22:51:59.643003 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637213    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 22:51:59.643003 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637216    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 22:51:59.643003 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637218    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 22:51:59.643003 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637221    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 22:51:59.643003 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637223    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 22:51:59.643003 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637226    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 22:51:59.643003 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637229    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 22:51:59.643497 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637231    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 22:51:59.643497 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637234    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 22:51:59.643497 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637244    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 22:51:59.643497 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637248    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 22:51:59.643497 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637250    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 22:51:59.643497 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637253    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 22:51:59.643497 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637256    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 22:51:59.643497 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637259    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 22:51:59.643497 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637261    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 22:51:59.643497 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637264    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 22:51:59.643497 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637267    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 22:51:59.643497 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637269    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 22:51:59.643497 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637272    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 22:51:59.643497 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637274    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 22:51:59.643497 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637277    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 22:51:59.643497 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637279    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 22:51:59.643497 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637282    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 22:51:59.643497 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637285    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 22:51:59.643497 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637287    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 22:51:59.643957 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637290    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 22:51:59.643957 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637292    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 22:51:59.643957 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637295    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 22:51:59.643957 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637297    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 22:51:59.643957 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637300    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 22:51:59.643957 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637303    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 22:51:59.643957 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637305    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 22:51:59.643957 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637308    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 22:51:59.643957 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637310    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 22:51:59.643957 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637313    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 22:51:59.643957 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637315    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 22:51:59.643957 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637318    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 22:51:59.643957 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637320    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 22:51:59.643957 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637324    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 22:51:59.643957 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637340    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 22:51:59.643957 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637343    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 22:51:59.643957 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637353    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 22:51:59.643957 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637356    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 22:51:59.643957 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637358    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 22:51:59.644491 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637361    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 22:51:59.644491 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637364    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 22:51:59.644491 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637366    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 22:51:59.644491 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637369    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 22:51:59.644491 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637372    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 22:51:59.644491 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637375    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 22:51:59.644491 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637377    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 22:51:59.644491 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637380    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 22:51:59.644491 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637383    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 22:51:59.644491 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637385    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 22:51:59.644491 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637388    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 22:51:59.644491 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637391    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 22:51:59.644491 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637395    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 22:51:59.644491 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637398    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 22:51:59.644491 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637401    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 22:51:59.644491 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637404    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 22:51:59.644491 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637407    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 22:51:59.644491 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637409    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 22:51:59.644491 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637412    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 22:51:59.644947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637414    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 22:51:59.644947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.637417    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 22:51:59.644947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.638522    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 22:51:59.644947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.638539    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 22:51:59.644947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.638549    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 22:51:59.644947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.638553    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 22:51:59.644947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.638556    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 22:51:59.644947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.638560    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 22:51:59.644947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.638564    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 22:51:59.644947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.638569    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 22:51:59.644947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.638574    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 22:51:59.644947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.638606    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 22:51:59.644947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.638612    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 22:51:59.644947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.638616    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 22:51:59.644947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.638620    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 22:51:59.644947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.638625    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 22:51:59.644947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.638629    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 22:51:59.644947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.638639    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 22:51:59.644947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.638653    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 22:51:59.644947 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.638659    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 22:51:59.645431 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.638728    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 22:51:59.645431 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.638739    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 22:51:59.645431 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.638922    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 22:51:59.645431 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639087    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 22:51:59.645431 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639093    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 22:51:59.645431 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639096    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 22:51:59.645431 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639099    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 22:51:59.645431 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639102    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 22:51:59.645431 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639105    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 22:51:59.645431 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639107    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 22:51:59.645431 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639110    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 22:51:59.645431 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639113    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 22:51:59.645431 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639138    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 22:51:59.645431 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639143    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 22:51:59.645431 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639146    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 22:51:59.645431 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639150    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 22:51:59.645817 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639154    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 22:51:59.645817 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639157    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 22:51:59.645817 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639159    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 22:51:59.645817 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639162    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 22:51:59.645817 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639164    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 22:51:59.645817 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639167    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 22:51:59.645817 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639170    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 22:51:59.645817 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639172    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 22:51:59.645817 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639175    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 22:51:59.645817 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639177    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 22:51:59.645817 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639181    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 22:51:59.645817 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639184    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 22:51:59.645817 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639187    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 22:51:59.645817 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639190    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 22:51:59.645817 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639192    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 22:51:59.645817 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639195    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 22:51:59.645817 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639198    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 22:51:59.645817 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639200    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 22:51:59.645817 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639203    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 22:51:59.646287 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639206    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 22:51:59.646287 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639209    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 22:51:59.646287 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639211    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 22:51:59.646287 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639214    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 22:51:59.646287 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639216    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 22:51:59.646287 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639219    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 22:51:59.646287 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639222    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 22:51:59.646287 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639224    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 22:51:59.646287 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639227    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 22:51:59.646287 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639230    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 22:51:59.646287 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639233    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 22:51:59.646287 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639238    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 22:51:59.646287 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639241    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 22:51:59.646287 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639244    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 22:51:59.646287 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639247    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 22:51:59.646287 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639249    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 22:51:59.646287 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639252    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 22:51:59.646287 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639254    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 22:51:59.646287 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639257    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 22:51:59.646749 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639260    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 22:51:59.646749 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639262    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 22:51:59.646749 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639265    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 22:51:59.646749 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639269    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 22:51:59.646749 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639271    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 22:51:59.646749 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639275    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 22:51:59.646749 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639277    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 22:51:59.646749 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639280    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 22:51:59.646749 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639282    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 22:51:59.646749 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639285    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 22:51:59.646749 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639289    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 22:51:59.646749 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639292    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 22:51:59.646749 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639295    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 22:51:59.646749 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639299    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 22:51:59.646749 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639302    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 22:51:59.646749 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639305    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 22:51:59.646749 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639307    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 22:51:59.646749 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639310    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 22:51:59.646749 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639313    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 22:51:59.647202 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639315    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 22:51:59.647202 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639318    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 22:51:59.647202 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639320    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 22:51:59.647202 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639323    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 22:51:59.647202 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639340    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 22:51:59.647202 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639345    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 22:51:59.647202 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639348    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 22:51:59.647202 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639361    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 22:51:59.647202 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639364    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 22:51:59.647202 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639366    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 22:51:59.647202 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639369    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 22:51:59.647202 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639372    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 22:51:59.647202 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639374    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 22:51:59.647202 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639377    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 22:51:59.647202 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639380    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 22:51:59.647202 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639383    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 22:51:59.647202 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639386    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 22:51:59.647202 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639388    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 22:51:59.647202 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639391    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 22:51:59.647202 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639394    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 22:51:59.647712 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639396    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 22:51:59.647712 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639399    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 22:51:59.647712 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639401    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 22:51:59.647712 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639404    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 22:51:59.647712 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639407    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 22:51:59.647712 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639410    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 22:51:59.647712 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639412    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 22:51:59.647712 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639415    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 22:51:59.647712 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639418    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 22:51:59.647712 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639420    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 22:51:59.647712 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639423    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 22:51:59.647712 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639425    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 22:51:59.647712 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639428    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 22:51:59.647712 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639430    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 22:51:59.647712 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639433    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 22:51:59.647712 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639436    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 22:51:59.648084 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.639461    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 22:51:59.648084 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639494    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 22:51:59.648084 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639498    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 22:51:59.648084 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639501    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 22:51:59.648084 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639504    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 22:51:59.648084 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639506    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 22:51:59.648084 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639509    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 22:51:59.648084 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639512    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 22:51:59.648084 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639514    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 22:51:59.648084 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639517    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 22:51:59.648084 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639519    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 22:51:59.648084 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639522    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 22:51:59.648084 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639525    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 22:51:59.648084 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639527    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 22:51:59.648084 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639530    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 22:51:59.648084 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639537    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 22:51:59.648483 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639540    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 22:51:59.648483 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639542    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 22:51:59.648483 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639545    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 22:51:59.648483 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639548    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 22:51:59.648483 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639550    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 22:51:59.648483 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639553    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 22:51:59.648483 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639564    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 22:51:59.648483 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639567    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 22:51:59.648483 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639571    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 22:51:59.648483 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639575    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 22:51:59.648483 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639579    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 22:51:59.648483 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639582    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 22:51:59.648483 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639584    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 22:51:59.648483 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639587    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 22:51:59.648483 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639590    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 22:51:59.648483 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639592    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 22:51:59.648483 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639595    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 22:51:59.648483 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639597    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 22:51:59.648483 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639600    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 22:51:59.648926 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639602    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 22:51:59.648926 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639605    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 22:51:59.648926 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639607    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 22:51:59.648926 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639610    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 22:51:59.648926 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639613    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 22:51:59.648926 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639615    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 22:51:59.648926 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639618    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 22:51:59.648926 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639620    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 22:51:59.648926 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639624    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 22:51:59.648926 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639628    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 22:51:59.648926 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639631    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 22:51:59.648926 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639634    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 22:51:59.648926 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639639    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 22:51:59.648926 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639642    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 22:51:59.648926 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639644    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 22:51:59.648926 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639647    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 22:51:59.648926 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639650    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 22:51:59.648926 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639652    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 22:51:59.648926 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639655    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 22:51:59.649469 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639657    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 22:51:59.649469 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639665    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 22:51:59.649469 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639674    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 22:51:59.649469 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639678    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 22:51:59.649469 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639681    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 22:51:59.649469 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639684    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 22:51:59.649469 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639686    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 22:51:59.649469 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639689    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 22:51:59.649469 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639692    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 22:51:59.649469 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639694    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 22:51:59.649469 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639697    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 22:51:59.649469 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639700    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 22:51:59.649469 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639702    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 22:51:59.649469 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639705    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 22:51:59.649469 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639708    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 22:51:59.649469 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639711    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 22:51:59.649469 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639713    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 22:51:59.649469 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639716    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 22:51:59.649469 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639718    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 22:51:59.649469 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639721    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 22:51:59.649942 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639723    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 22:51:59.649942 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639726    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 22:51:59.649942 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639728    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 22:51:59.649942 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639731    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 22:51:59.649942 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639734    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 22:51:59.649942 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639739    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 22:51:59.649942 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639742    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 22:51:59.649942 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639744    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 22:51:59.649942 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639747    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 22:51:59.649942 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639750    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 22:51:59.649942 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639752    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 22:51:59.649942 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639755    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 22:51:59.649942 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639757    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 22:51:59.649942 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639760    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 22:51:59.649942 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639769    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 22:51:59.649942 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639772    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 22:51:59.649942 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639775    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 22:51:59.649942 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639777    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 22:51:59.649942 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639780    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 22:51:59.649942 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639783    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 22:51:59.650428 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639785    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 22:51:59.650428 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639788    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 22:51:59.650428 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639790    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 22:51:59.650428 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639793    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 22:51:59.650428 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639795    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 22:51:59.650428 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639798    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 22:51:59.650428 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639801    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 22:51:59.650428 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639804    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 22:51:59.650428 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639806    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 22:51:59.650428 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639809    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 22:51:59.650428 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639812    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 22:51:59.650428 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639814    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 22:51:59.650428 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:51:59.639817    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 22:51:59.650428 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.639895    2582 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 22:51:59.650428 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.641824    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 22:51:59.650428 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.642658    2582 server.go:1014] "Starting client certificate rotation"
Nov 02 22:51:59.650428 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.642749    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 22:51:59.650428 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.643464    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 22:51:59.650879 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.644473    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 22:51:59.650879 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.645607    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 22:51:59.654962 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.654945    2582 log.go:25] "Validated CRI v1 runtime API"
Nov 02 22:51:59.658318 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.658300    2582 log.go:25] "Validated CRI v1 image API"
Nov 02 22:51:59.659186 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.659158    2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 22:51:59.663301 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.663284    2582 fs.go:135] Filesystem UUIDs: map[3c7acda2-d5e0-49c6-afa4-2d41cb8a711a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e93a8b64-fb10-4893-a847-7c0dcf7ceca1:/dev/nvme0n1p3]
Nov 02 22:51:59.663370 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.663301    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 22:51:59.669903 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.669785    2582 manager.go:217] Machine: {Timestamp:2026-11-02 22:51:59.668375216 +0000 UTC m=+0.296793199 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096199 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dde3fe5e593041dc59bf425971cf8 SystemUUID:ec2dde3f-e5e5-9304-1dc5-9bf425971cf8 BootID:1b9479a7-4fc0-4a45-a5b4-65f211d57cdd Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:e4:77:29:ab Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:e4:77:29:ab Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:6d:82:40:71:90 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 22:51:59.669903 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.669900    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 22:51:59.670030 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.670015    2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 22:51:59.672503 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.672478    2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 22:51:59.672674 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.672505    2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-234.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 22:51:59.672718 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.672677    2582 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 22:51:59.672718 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.672685    2582 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 22:51:59.672718 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.672716    2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 22:51:59.672796 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.672735    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 22:51:59.673353 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.673342    2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 22:51:59.677051 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.677038    2582 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 22:51:59.677177 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.677168    2582 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 22:51:59.677834 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.677825    2582 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 22:51:59.677865 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.677838    2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 22:51:59.677865 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.677849    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 22:51:59.677865 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.677858    2582 kubelet.go:405] "Adding apiserver pod source"
Nov 02 22:51:59.677945 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.677866    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 22:51:59.678987 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.678962    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 22:51:59.682033 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.682010    2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 22:51:59.685800 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.685786    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 22:51:59.686477 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.686464    2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 22:51:59.686525 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.686495    2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 22:51:59.686672 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.686661    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 22:51:59.686723 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.686678    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 22:51:59.686723 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.686685    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 22:51:59.686723 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.686691    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 22:51:59.686723 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.686696    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 22:51:59.686723 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.686703    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 22:51:59.687361 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.687350    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 22:51:59.687361 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.687359    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 22:51:59.687452 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.687367    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 22:51:59.687452 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.687373    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 22:51:59.687452 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.687382    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 22:51:59.687452 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.687391    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 22:51:59.687933 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.687922    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 22:51:59.687970 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.687936    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 22:51:59.691496 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.691485    2582 server.go:1263] "Started kubelet"
Nov 02 22:51:59.691730 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.691698    2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 22:51:59.692211 ip-10-0-139-234 systemd[1]: Started Kubernetes Kubelet.
Nov 02 22:51:59.693278 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.692260    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 22:51:59.693278 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.692354    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 22:51:59.693278 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.693240    2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 22:51:59.694392 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.694373    2582 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 22:51:59.697446 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:51:59.697428    2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 22:51:59.698994 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.698975    2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 22:51:59.699068 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.698992    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 22:51:59.699243 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:51:59.699213    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-234.us-west-2.compute.internal\" not found"
Nov 02 22:51:59.699861 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.699818    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 22:51:59.700670 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.700652    2582 factory.go:153] Registering CRI-O factory
Nov 02 22:51:59.700670 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.700669    2582 factory.go:223] Registration of the crio container factory successfully
Nov 02 22:51:59.700810 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.700707    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 22:51:59.700810 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.700713    2582 factory.go:55] Registering systemd factory
Nov 02 22:51:59.700810 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.700720    2582 factory.go:223] Registration of the systemd container factory successfully
Nov 02 22:51:59.700810 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.700736    2582 factory.go:103] Registering Raw factory
Nov 02 22:51:59.700810 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.700746    2582 manager.go:1199] Started watching for new ooms in manager
Nov 02 22:51:59.701780 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.701762    2582 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 22:51:59.701780 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.701781    2582 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 22:51:59.701939 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.701911    2582 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 22:51:59.701939 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.701921    2582 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 22:51:59.702039 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.702024    2582 manager.go:319] Starting recovery of all containers
Nov 02 22:51:59.714701 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.714681    2582 manager.go:324] Recovery completed
Nov 02 22:51:59.718782 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.718768    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 22:51:59.720536 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.720518    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 22:51:59.720608 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.720548    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 22:51:59.720608 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.720558    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 22:51:59.721090 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.721080    2582 cpu_manager.go:226] "Starting" policy="none"
Nov 02 22:51:59.721123 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.721088    2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 22:51:59.721123 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.721094    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 22:51:59.721123 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.721105    2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 22:51:59.721791 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.721776    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 22:51:59.721833 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.721806    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 22:51:59.721833 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.721819    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 22:51:59.723392 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.723375    2582 policy_none.go:50] "Start"
Nov 02 22:51:59.723465 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.723395    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 22:51:59.723465 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.723409    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 22:51:59.725393 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.725382    2582 policy_none.go:44] "Start"
Nov 02 22:51:59.748938 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.748910    2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 22:51:59.753706 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.753690    2582 manager.go:341] "Starting Device Plugin manager"
Nov 02 22:51:59.753842 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:51:59.753725    2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 22:51:59.753842 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.753737    2582 server.go:87] "Starting device plugin registration server"
Nov 02 22:51:59.754030 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.754018    2582 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 22:51:59.754072 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.754032    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 22:51:59.754123 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.754109    2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 22:51:59.754217 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.754204    2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 22:51:59.754283 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.754231    2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 22:51:59.755095 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:51:59.755075    2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 22:51:59.755186 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:51:59.755123    2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-234.us-west-2.compute.internal\" not found"
Nov 02 22:51:59.840033 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.840002    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 22:51:59.841169 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.841147    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 22:51:59.841262 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.841178    2582 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 22:51:59.841262 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.841206    2582 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 22:51:59.841262 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:51:59.841246    2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 22:51:59.842453 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.842435    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-234.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 22:51:59.842453 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:51:59.842428    2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-234.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 22:51:59.843398 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:51:59.842423    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-234.us-west-2.compute.internal.18e45c3199109358  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-234.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-234.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 22:51:59.691453272 +0000 UTC m=+0.319871256,LastTimestamp:2026-11-02 22:51:59.691453272 +0000 UTC m=+0.319871256,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-234.us-west-2.compute.internal,}"
Nov 02 22:51:59.855023 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.855006    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 22:51:59.855839 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.855818    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 22:51:59.855943 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.855851    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 22:51:59.855943 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.855862    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 22:51:59.855943 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.855887    2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:51:59.900470 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.900251    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 22:51:59.900470 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.900303    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fdpwj"
Nov 02 22:51:59.942366 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.942315    2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal"]
Nov 02 22:51:59.942477 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.942402    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 22:51:59.943418 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.943400    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 22:51:59.943513 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.943430    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 22:51:59.943513 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.943442    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 22:51:59.945711 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.945697    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 22:51:59.945863 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.945845    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:51:59.945901 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.945881    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 22:51:59.946572 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.946553    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 22:51:59.946693 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.946584    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 22:51:59.946693 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.946599    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 22:51:59.946693 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.946554    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 22:51:59.946693 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.946652    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 22:51:59.946693 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.946662    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 22:51:59.950158 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.950134    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:51:59.950241 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.950185    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 22:51:59.951315 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.951296    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 22:51:59.951418 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.951344    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 22:51:59.951418 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.951377    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 22:51:59.969779 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:51:59.969760    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-234.us-west-2.compute.internal\" not found" node="ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:51:59.971917 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:51:59.971901    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-234.us-west-2.compute.internal\" not found" node="ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:51:59.999048 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:51:59.999029    2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 22:52:00.003994 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:00.003975    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a401e1afc8eeebc2962b0eea90d39ec6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal\" (UID: \"a401e1afc8eeebc2962b0eea90d39ec6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:00.004069 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:00.004002    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/990ba2ba1d2d380f95d805ba47d10a19-config\") pod \"kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal\" (UID: \"990ba2ba1d2d380f95d805ba47d10a19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:00.004069 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:00.004019    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a401e1afc8eeebc2962b0eea90d39ec6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal\" (UID: \"a401e1afc8eeebc2962b0eea90d39ec6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:00.094039 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:00.094010    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 22:52:00.094039 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:00.094010    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 22:52:00.094692 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:00.094677    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 22:52:00.095021 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:00.094993    2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:00.095021 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:00.095020    2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-234.us-west-2.compute.internal\": node \"ip-10-0-139-234.us-west-2.compute.internal\" not found"
Nov 02 22:52:00.099143 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:00.099126    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:00.104119 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:00.104102    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a401e1afc8eeebc2962b0eea90d39ec6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal\" (UID: \"a401e1afc8eeebc2962b0eea90d39ec6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:00.104201 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:00.104126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a401e1afc8eeebc2962b0eea90d39ec6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal\" (UID: \"a401e1afc8eeebc2962b0eea90d39ec6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:00.104201 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:00.104145    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/990ba2ba1d2d380f95d805ba47d10a19-config\") pod \"kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal\" (UID: \"990ba2ba1d2d380f95d805ba47d10a19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:00.104269 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:00.104200    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/990ba2ba1d2d380f95d805ba47d10a19-config\") pod \"kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal\" (UID: \"990ba2ba1d2d380f95d805ba47d10a19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:00.104269 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:00.104202    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a401e1afc8eeebc2962b0eea90d39ec6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal\" (UID: \"a401e1afc8eeebc2962b0eea90d39ec6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:00.104269 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:00.104208    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a401e1afc8eeebc2962b0eea90d39ec6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal\" (UID: \"a401e1afc8eeebc2962b0eea90d39ec6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:00.158010 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:00.157955    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 22:52:00.158109 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:00.158050    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:00.213295 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:00.213274    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 22:52:00.272408 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:00.272387    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:00.275274 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:00.275260    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:02.158077 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.158052    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 22:52:02.158545 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.158252    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 22:52:02.158545 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.158259    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 22:52:02.158545 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:02.158258    2582 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a2b73d5d16d574ed99182329d58f278a-a10e0d37d7b270df.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-139-234.us-west-2.compute.internal.18e45c319acc522a\": read tcp 10.0.139.234:40878->52.14.154.205:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-139-234.us-west-2.compute.internal.18e45c319acc522a  default   5150 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-234.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-234.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-234.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 22:51:59 +0000 UTC,LastTimestamp:2026-11-02 22:51:59.94341623 +0000 UTC m=+0.571834214,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-234.us-west-2.compute.internal,}"
Nov 02 22:52:02.194111 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.194078    2582 apiserver.go:52] "Watching apiserver"
Nov 02 22:52:02.214908 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.214885    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 22:52:02.420531 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.420439    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 22:47:01 +0000 UTC" deadline="2028-07-06 01:28:17.646192584 +0000 UTC"
Nov 02 22:52:02.420531 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.420476    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14666h36m15.225719362s"
Nov 02 22:52:02.450224 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.450199    2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 22:52:02.451125 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.451090    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-295c4","kube-system/konnectivity-agent-6zx4w","kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-6gn7z","openshift-cluster-node-tuning-operator/tuned-smds5","openshift-multus/multus-additional-cni-plugins-kbhxn","kube-system/global-pull-secret-syncer-hlfzd","openshift-dns/node-resolver-r2vkn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal","openshift-multus/multus-q5cp2","openshift-network-operator/iptables-alerter-p5hzg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl","openshift-image-registry/node-ca-hfl65","openshift-network-diagnostics/network-check-target-qhbsn"]
Nov 02 22:52:02.453892 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.453872    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.455993 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.455965    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6zx4w"
Nov 02 22:52:02.458038 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.458018    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:02.458117 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:02.458089    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gn7z" podUID="4b1cf92b-28eb-4894-96ed-b7eba4a781d2"
Nov 02 22:52:02.460122 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.460108    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.462243 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.462231    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.464360 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.464345    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:02.464426 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:02.464401    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hlfzd" podUID="3a474267-46b1-4759-9018-54960f751878"
Nov 02 22:52:02.466525 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.466508    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r2vkn"
Nov 02 22:52:02.468734 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.468717    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.471053 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.471038    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p5hzg"
Nov 02 22:52:02.473210 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.473194    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.475548 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.475534    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hfl65"
Nov 02 22:52:02.477648 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.477631    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:02.477713 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:02.477691    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhbsn" podUID="8161cb48-56bb-455b-bee6-cd3d06607109"
Nov 02 22:52:02.504226 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.504209    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 22:52:02.504482 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.504455    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 22:52:02.504482 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.504483    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 22:52:02.504643 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.504496    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 22:52:02.504724 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.504710    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 22:52:02.504765 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.504711    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c9lzr\""
Nov 02 22:52:02.504797 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.504779    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 22:52:02.505269 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.505254    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 22:52:02.505543 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.505531    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-crtkq\""
Nov 02 22:52:02.505717 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.505703    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 22:52:02.509463 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.509448    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 22:52:02.509523 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.509452    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 22:52:02.517188 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.517174    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 22:52:02.525145 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.525128    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-etc-selinux\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.525220 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.525153    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-lib-modules\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.525220 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.525171    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1bd54f8-715f-428f-998b-65c6e7f8830e-cni-binary-copy\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.525220 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.525193    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1bd54f8-715f-428f-998b-65c6e7f8830e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.525662 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.525532    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-run-netns\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.525662 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.525591    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a474267-46b1-4759-9018-54960f751878-kubelet-config\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:02.525788 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.525660    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hl8b\" (UniqueName: \"kubernetes.io/projected/b1bd54f8-715f-428f-998b-65c6e7f8830e-kube-api-access-5hl8b\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.525788 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.525705    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-cnibin\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.525788 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.525739    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-var-lib-kubelet\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.525788 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.525769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-run-ovn-kubernetes\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.525966 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.525802    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-var-lib-kubelet\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.525966 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.525826    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-host\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.525966 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.525855    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1bd54f8-715f-428f-998b-65c6e7f8830e-cnibin\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.525966 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.525885    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-ovnkube-script-lib\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.525966 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.525917    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-456qc\" (UniqueName: \"kubernetes.io/projected/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-kube-api-access-456qc\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.525966 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.525955    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0dfa862-aaf8-4ff1-820e-a610a4d41ba8-konnectivity-ca\") pod \"konnectivity-agent-6zx4w\" (UID: \"a0dfa862-aaf8-4ff1-820e-a610a4d41ba8\") " pod="kube-system/konnectivity-agent-6zx4w"
Nov 02 22:52:02.526235 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526006    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-socket-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.526235 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526035    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-tuned\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.526235 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526076    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18dcacc0-751d-4f2e-8101-59a1da48fab2-tmp\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.526235 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526103    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-socket-dir-parent\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.526235 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526126    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-var-lib-cni-bin\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.526235 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-run-openvswitch\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.526235 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526198    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-modprobe-d\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.526235 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526228    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-run-ovn\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.526560 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526256    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-cni-bin\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.526560 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526282    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-hostroot\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.526560 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526308    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6hww\" (UniqueName: \"kubernetes.io/projected/0ce80856-f167-427e-b8d5-4b856bfcf1ea-kube-api-access-r6hww\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.526560 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526335    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-var-lib-openvswitch\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.526560 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-cni-netd\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.526560 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526391    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-system-cni-dir\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.526560 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526423    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-slash\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.526560 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526446    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-ovnkube-config\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.526560 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526477    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-sys-fs\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.526560 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526505    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-sysctl-conf\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.526560 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526534    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9zml\" (UniqueName: \"kubernetes.io/projected/18dcacc0-751d-4f2e-8101-59a1da48fab2-kube-api-access-g9zml\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.526560 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526562    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-conf-dir\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.527074 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526584    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-run\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.527074 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526611    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1bd54f8-715f-428f-998b-65c6e7f8830e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.527074 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526641    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-run-multus-certs\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.527074 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526670    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e37c6a5d-1170-4f2d-ae18-2580da36f53b-host-slash\") pod \"iptables-alerter-p5hzg\" (UID: \"e37c6a5d-1170-4f2d-ae18-2580da36f53b\") " pod="openshift-network-operator/iptables-alerter-p5hzg"
Nov 02 22:52:02.527074 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526703    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-run-systemd\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.527074 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526729    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-log-socket\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.527074 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526756    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs\") pod \"network-metrics-daemon-6gn7z\" (UID: \"4b1cf92b-28eb-4894-96ed-b7eba4a781d2\") " pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:02.527074 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526783    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a474267-46b1-4759-9018-54960f751878-dbus\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:02.527074 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526815    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.527074 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526843    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxxqd\" (UniqueName: \"kubernetes.io/projected/11efd367-1b70-4cb5-93c5-2dda7b9bb010-kube-api-access-nxxqd\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.527074 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526870    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-sysctl-d\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.527074 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526895    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e37c6a5d-1170-4f2d-ae18-2580da36f53b-iptables-alerter-script\") pod \"iptables-alerter-p5hzg\" (UID: \"e37c6a5d-1170-4f2d-ae18-2580da36f53b\") " pod="openshift-network-operator/iptables-alerter-p5hzg"
Nov 02 22:52:02.527074 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526924    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drxcv\" (UniqueName: \"kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv\") pod \"network-check-target-qhbsn\" (UID: \"8161cb48-56bb-455b-bee6-cd3d06607109\") " pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:02.527074 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526951    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92845990-417f-4c8f-aa67-ea9961516288-hosts-file\") pod \"node-resolver-r2vkn\" (UID: \"92845990-417f-4c8f-aa67-ea9961516288\") " pod="openshift-dns/node-resolver-r2vkn"
Nov 02 22:52:02.527074 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.526992    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-run-netns\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.527074 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-env-overrides\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.527746 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527045    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-ovn-node-metrics-cert\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.527746 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527076    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0dfa862-aaf8-4ff1-820e-a610a4d41ba8-agent-certs\") pod \"konnectivity-agent-6zx4w\" (UID: \"a0dfa862-aaf8-4ff1-820e-a610a4d41ba8\") " pod="kube-system/konnectivity-agent-6zx4w"
Nov 02 22:52:02.527746 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527107    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4nhs\" (UniqueName: \"kubernetes.io/projected/92845990-417f-4c8f-aa67-ea9961516288-kube-api-access-c4nhs\") pod \"node-resolver-r2vkn\" (UID: \"92845990-417f-4c8f-aa67-ea9961516288\") " pod="openshift-dns/node-resolver-r2vkn"
Nov 02 22:52:02.527746 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527136    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ce80856-f167-427e-b8d5-4b856bfcf1ea-cni-binary-copy\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.527746 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527160    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-var-lib-cni-multus\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.527746 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527189    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9qmn\" (UniqueName: \"kubernetes.io/projected/e37c6a5d-1170-4f2d-ae18-2580da36f53b-kube-api-access-j9qmn\") pod \"iptables-alerter-p5hzg\" (UID: \"e37c6a5d-1170-4f2d-ae18-2580da36f53b\") " pod="openshift-network-operator/iptables-alerter-p5hzg"
Nov 02 22:52:02.527746 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527217    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-systemd-units\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.527746 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527247    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25fvj\" (UniqueName: \"kubernetes.io/projected/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-kube-api-access-25fvj\") pod \"network-metrics-daemon-6gn7z\" (UID: \"4b1cf92b-28eb-4894-96ed-b7eba4a781d2\") " pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:02.527746 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527276    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e-host\") pod \"node-ca-hfl65\" (UID: \"8e71e09c-79f2-42d0-a2ee-e1535cf3df8e\") " pod="openshift-image-registry/node-ca-hfl65"
Nov 02 22:52:02.527746 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527299    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-sysconfig\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.527746 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527336    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-systemd\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.527746 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1bd54f8-715f-428f-998b-65c6e7f8830e-system-cni-dir\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.527746 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527393    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-cni-dir\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.527746 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527422    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-daemon-config\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.527746 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527450    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-kubelet\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.527746 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527479    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-device-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.528310 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527507    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-kubernetes\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.528310 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527535    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-sys\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.528310 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527564    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-etc-kubernetes\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.528310 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527593    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.528310 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527623    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:02.528310 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527651    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92845990-417f-4c8f-aa67-ea9961516288-tmp-dir\") pod \"node-resolver-r2vkn\" (UID: \"92845990-417f-4c8f-aa67-ea9961516288\") " pod="openshift-dns/node-resolver-r2vkn"
Nov 02 22:52:02.528310 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527688    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e-serviceca\") pod \"node-ca-hfl65\" (UID: \"8e71e09c-79f2-42d0-a2ee-e1535cf3df8e\") " pod="openshift-image-registry/node-ca-hfl65"
Nov 02 22:52:02.528310 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527713    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9m47\" (UniqueName: \"kubernetes.io/projected/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e-kube-api-access-x9m47\") pod \"node-ca-hfl65\" (UID: \"8e71e09c-79f2-42d0-a2ee-e1535cf3df8e\") " pod="openshift-image-registry/node-ca-hfl65"
Nov 02 22:52:02.528310 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1bd54f8-715f-428f-998b-65c6e7f8830e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.528310 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.527900    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-node-log\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.528310 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.528059    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-registration-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.528310 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.528130    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-run-k8s-cni-cncf-io\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.528310 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.528218    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-etc-openvswitch\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.547211 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.547194    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 22:52:02.547302 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.547229    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 22:52:02.547302 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.547238    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9p98l\""
Nov 02 22:52:02.547302 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.547287    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-92nqq\""
Nov 02 22:52:02.551251 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.551228    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 22:52:02.551251 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.551234    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 22:52:02.551385 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.551255    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 22:52:02.551385 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.551277    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-v5hvl\""
Nov 02 22:52:02.552007 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.551970    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 22:52:02.552115 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.552024    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 22:52:02.552115 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.552062    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rxqrv\""
Nov 02 22:52:02.552115 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.552066    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 22:52:02.552115 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.552088    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9qgnp\""
Nov 02 22:52:02.552465 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.552449    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 22:52:02.552536 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.552479    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 22:52:02.552536 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.552480    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 22:52:02.552536 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.552498    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 22:52:02.552536 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.552502    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 22:52:02.552653 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.552540    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 22:52:02.552653 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.552557    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 22:52:02.552653 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.552579    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 22:52:02.552653 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.552599    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 22:52:02.553063 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.553046    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sdz5j\""
Nov 02 22:52:02.553063 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.553062    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-77brg\""
Nov 02 22:52:02.553314 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.553295    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 22:52:02.553401 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.553344    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-trpms"
Nov 02 22:52:02.628741 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.628712    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-run\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.628741 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.628741    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1bd54f8-715f-428f-998b-65c6e7f8830e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.628936 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.628757    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-run-multus-certs\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.628936 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.628771    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e37c6a5d-1170-4f2d-ae18-2580da36f53b-host-slash\") pod \"iptables-alerter-p5hzg\" (UID: \"e37c6a5d-1170-4f2d-ae18-2580da36f53b\") " pod="openshift-network-operator/iptables-alerter-p5hzg"
Nov 02 22:52:02.628936 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.628789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-run-systemd\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.628936 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.628812    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-log-socket\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.628936 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.628851    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-run-multus-certs\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.628936 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.628855    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs\") pod \"network-metrics-daemon-6gn7z\" (UID: \"4b1cf92b-28eb-4894-96ed-b7eba4a781d2\") " pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:02.628936 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.628872    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e37c6a5d-1170-4f2d-ae18-2580da36f53b-host-slash\") pod \"iptables-alerter-p5hzg\" (UID: \"e37c6a5d-1170-4f2d-ae18-2580da36f53b\") " pod="openshift-network-operator/iptables-alerter-p5hzg"
Nov 02 22:52:02.628936 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.628889    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-run-systemd\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.628936 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.628899    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a474267-46b1-4759-9018-54960f751878-dbus\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:02.628936 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.628909    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-run\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.628936 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.628917    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1bd54f8-715f-428f-998b-65c6e7f8830e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.628936 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.628924    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.628936 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.628917    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-log-socket\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.629597 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:02.628958    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 22:52:02.629597 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.628960    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a474267-46b1-4759-9018-54960f751878-dbus\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:02.629597 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.628964    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxxqd\" (UniqueName: \"kubernetes.io/projected/11efd367-1b70-4cb5-93c5-2dda7b9bb010-kube-api-access-nxxqd\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.629597 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.628998    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.629597 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629011    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-sysctl-d\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.629597 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:02.629043    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs podName:4b1cf92b-28eb-4894-96ed-b7eba4a781d2 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:03.129013636 +0000 UTC m=+2.241781645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs") pod "network-metrics-daemon-6gn7z" (UID: "4b1cf92b-28eb-4894-96ed-b7eba4a781d2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 22:52:02.629597 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629071    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e37c6a5d-1170-4f2d-ae18-2580da36f53b-iptables-alerter-script\") pod \"iptables-alerter-p5hzg\" (UID: \"e37c6a5d-1170-4f2d-ae18-2580da36f53b\") " pod="openshift-network-operator/iptables-alerter-p5hzg"
Nov 02 22:52:02.629597 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629090    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-sysctl-d\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.629597 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxcv\" (UniqueName: \"kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv\") pod \"network-check-target-qhbsn\" (UID: \"8161cb48-56bb-455b-bee6-cd3d06607109\") " pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:02.629597 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92845990-417f-4c8f-aa67-ea9961516288-hosts-file\") pod \"node-resolver-r2vkn\" (UID: \"92845990-417f-4c8f-aa67-ea9961516288\") " pod="openshift-dns/node-resolver-r2vkn"
Nov 02 22:52:02.629597 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629149    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-run-netns\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.629597 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629174    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-env-overrides\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.629597 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629185    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92845990-417f-4c8f-aa67-ea9961516288-hosts-file\") pod \"node-resolver-r2vkn\" (UID: \"92845990-417f-4c8f-aa67-ea9961516288\") " pod="openshift-dns/node-resolver-r2vkn"
Nov 02 22:52:02.629597 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629195    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-ovn-node-metrics-cert\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.629597 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629218    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-run-netns\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.629597 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629218    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0dfa862-aaf8-4ff1-820e-a610a4d41ba8-agent-certs\") pod \"konnectivity-agent-6zx4w\" (UID: \"a0dfa862-aaf8-4ff1-820e-a610a4d41ba8\") " pod="kube-system/konnectivity-agent-6zx4w"
Nov 02 22:52:02.629597 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629255    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4nhs\" (UniqueName: \"kubernetes.io/projected/92845990-417f-4c8f-aa67-ea9961516288-kube-api-access-c4nhs\") pod \"node-resolver-r2vkn\" (UID: \"92845990-417f-4c8f-aa67-ea9961516288\") " pod="openshift-dns/node-resolver-r2vkn"
Nov 02 22:52:02.630298 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629279    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ce80856-f167-427e-b8d5-4b856bfcf1ea-cni-binary-copy\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.630298 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629314    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-var-lib-cni-multus\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.630298 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629338    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9qmn\" (UniqueName: \"kubernetes.io/projected/e37c6a5d-1170-4f2d-ae18-2580da36f53b-kube-api-access-j9qmn\") pod \"iptables-alerter-p5hzg\" (UID: \"e37c6a5d-1170-4f2d-ae18-2580da36f53b\") " pod="openshift-network-operator/iptables-alerter-p5hzg"
Nov 02 22:52:02.630298 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629367    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-systemd-units\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.630298 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629392    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25fvj\" (UniqueName: \"kubernetes.io/projected/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-kube-api-access-25fvj\") pod \"network-metrics-daemon-6gn7z\" (UID: \"4b1cf92b-28eb-4894-96ed-b7eba4a781d2\") " pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:02.630298 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629415    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e-host\") pod \"node-ca-hfl65\" (UID: \"8e71e09c-79f2-42d0-a2ee-e1535cf3df8e\") " pod="openshift-image-registry/node-ca-hfl65"
Nov 02 22:52:02.630298 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629437    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-sysconfig\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.630298 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629458    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-systemd\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.630298 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629488    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1bd54f8-715f-428f-998b-65c6e7f8830e-system-cni-dir\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.630298 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629473    2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 22:52:02.630298 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629526    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-cni-dir\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.630298 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629554    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-daemon-config\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.630298 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629575    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-kubelet\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.630298 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629598    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-device-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.630298 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-kubernetes\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.630298 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629626    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-systemd-units\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.630298 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629648    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-sys\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.630298 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629652    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-env-overrides\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.630954 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629674    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-etc-kubernetes\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.630954 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629697    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-cni-dir\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.630954 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629698    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.630954 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629712    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e37c6a5d-1170-4f2d-ae18-2580da36f53b-iptables-alerter-script\") pod \"iptables-alerter-p5hzg\" (UID: \"e37c6a5d-1170-4f2d-ae18-2580da36f53b\") " pod="openshift-network-operator/iptables-alerter-p5hzg"
Nov 02 22:52:02.630954 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629746    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1bd54f8-715f-428f-998b-65c6e7f8830e-system-cni-dir\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.630954 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629756    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.630954 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629764    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-etc-kubernetes\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.630954 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629778    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-sysconfig\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.630954 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629792    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-kubelet\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.630954 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629800    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-var-lib-cni-multus\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.630954 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629804    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-sys\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.630954 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629817    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:02.630954 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629835    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e-host\") pod \"node-ca-hfl65\" (UID: \"8e71e09c-79f2-42d0-a2ee-e1535cf3df8e\") " pod="openshift-image-registry/node-ca-hfl65"
Nov 02 22:52:02.630954 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629870    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-kubernetes\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.630954 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629882    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92845990-417f-4c8f-aa67-ea9961516288-tmp-dir\") pod \"node-resolver-r2vkn\" (UID: \"92845990-417f-4c8f-aa67-ea9961516288\") " pod="openshift-dns/node-resolver-r2vkn"
Nov 02 22:52:02.630954 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e-serviceca\") pod \"node-ca-hfl65\" (UID: \"8e71e09c-79f2-42d0-a2ee-e1535cf3df8e\") " pod="openshift-image-registry/node-ca-hfl65"
Nov 02 22:52:02.630954 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629920    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-systemd\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.630954 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:02.629928    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 22:52:02.631720 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:02.629993    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret podName:3a474267-46b1-4759-9018-54960f751878 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:03.129962902 +0000 UTC m=+2.242730914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret") pod "global-pull-secret-syncer-hlfzd" (UID: "3a474267-46b1-4759-9018-54960f751878") : object "kube-system"/"original-pull-secret" not registered
Nov 02 22:52:02.631720 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.629971    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ce80856-f167-427e-b8d5-4b856bfcf1ea-cni-binary-copy\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.631720 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630040    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-device-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.631720 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9m47\" (UniqueName: \"kubernetes.io/projected/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e-kube-api-access-x9m47\") pod \"node-ca-hfl65\" (UID: \"8e71e09c-79f2-42d0-a2ee-e1535cf3df8e\") " pod="openshift-image-registry/node-ca-hfl65"
Nov 02 22:52:02.631720 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630121    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1bd54f8-715f-428f-998b-65c6e7f8830e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.631720 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630159    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-node-log\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.631720 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-registration-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.631720 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-run-k8s-cni-cncf-io\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.631720 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630292    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-run-k8s-cni-cncf-io\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.631720 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630302    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-node-log\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.631720 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630330    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-registration-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.631720 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630345    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-etc-openvswitch\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.631720 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630388    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-etc-openvswitch\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.631720 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630415    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-etc-selinux\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.631720 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630438    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-lib-modules\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.631720 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630458    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1bd54f8-715f-428f-998b-65c6e7f8830e-cni-binary-copy\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.632514 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630479    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1bd54f8-715f-428f-998b-65c6e7f8830e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.632514 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630488    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-etc-selinux\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.632514 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630532    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-lib-modules\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.632514 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630564    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-run-netns\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.632514 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630586    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a474267-46b1-4759-9018-54960f751878-kubelet-config\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:02.632514 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630608    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hl8b\" (UniqueName: \"kubernetes.io/projected/b1bd54f8-715f-428f-998b-65c6e7f8830e-kube-api-access-5hl8b\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.632514 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630629    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-cnibin\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.632514 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630631    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-daemon-config\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.632514 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630649    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-var-lib-kubelet\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.632514 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630664    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a474267-46b1-4759-9018-54960f751878-kubelet-config\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:02.632514 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630672    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-run-ovn-kubernetes\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.632514 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630697    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-var-lib-kubelet\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.632514 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630718    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-host\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.632514 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1bd54f8-715f-428f-998b-65c6e7f8830e-cnibin\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.632514 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630744    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92845990-417f-4c8f-aa67-ea9961516288-tmp-dir\") pod \"node-resolver-r2vkn\" (UID: \"92845990-417f-4c8f-aa67-ea9961516288\") " pod="openshift-dns/node-resolver-r2vkn"
Nov 02 22:52:02.632514 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630765    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-ovnkube-script-lib\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.632514 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630790    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-run-netns\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.633350 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630808    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-456qc\" (UniqueName: \"kubernetes.io/projected/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-kube-api-access-456qc\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.633350 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630838    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0dfa862-aaf8-4ff1-820e-a610a4d41ba8-konnectivity-ca\") pod \"konnectivity-agent-6zx4w\" (UID: \"a0dfa862-aaf8-4ff1-820e-a610a4d41ba8\") " pod="kube-system/konnectivity-agent-6zx4w"
Nov 02 22:52:02.633350 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630847    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-cnibin\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.633350 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630863    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-socket-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.633350 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630864    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e-serviceca\") pod \"node-ca-hfl65\" (UID: \"8e71e09c-79f2-42d0-a2ee-e1535cf3df8e\") " pod="openshift-image-registry/node-ca-hfl65"
Nov 02 22:52:02.633350 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630880    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-var-lib-kubelet\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.633350 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630898    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-host\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.633350 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630902    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-var-lib-kubelet\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.633350 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630935    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-run-ovn-kubernetes\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.633350 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.630939    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1bd54f8-715f-428f-998b-65c6e7f8830e-cnibin\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.633350 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631082    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-socket-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.633350 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631175    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-tuned\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.633350 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631213    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18dcacc0-751d-4f2e-8101-59a1da48fab2-tmp\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.633350 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-socket-dir-parent\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.633350 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-var-lib-cni-bin\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.633350 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631289    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-run-openvswitch\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.633350 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631312    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-modprobe-d\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.633350 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631344    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-run-ovn\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.633885 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631388    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-cni-bin\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.633885 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631412    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-hostroot\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.633885 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631417    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-ovnkube-script-lib\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.633885 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631437    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6hww\" (UniqueName: \"kubernetes.io/projected/0ce80856-f167-427e-b8d5-4b856bfcf1ea-kube-api-access-r6hww\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.633885 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631464    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-var-lib-openvswitch\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.633885 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631484    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0dfa862-aaf8-4ff1-820e-a610a4d41ba8-konnectivity-ca\") pod \"konnectivity-agent-6zx4w\" (UID: \"a0dfa862-aaf8-4ff1-820e-a610a4d41ba8\") " pod="kube-system/konnectivity-agent-6zx4w"
Nov 02 22:52:02.633885 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631486    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-cni-netd\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.633885 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631533    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-system-cni-dir\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.633885 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631538    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-var-lib-cni-bin\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.633885 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631561    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-slash\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.633885 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631587    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-ovnkube-config\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.633885 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631591    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-socket-dir-parent\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.633885 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631608    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-cni-bin\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.633885 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631636    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-hostroot\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.633885 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631644    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-slash\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.633885 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631646    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-system-cni-dir\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.633885 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631658    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-sys-fs\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.633885 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631612    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-sys-fs\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.634380 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631694    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-cni-netd\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.634380 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631773    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-sysctl-conf\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.634380 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631804    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-modprobe-d\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.634380 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631834    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-run-ovn\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.634380 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631856    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-run-openvswitch\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.634380 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631885    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-var-lib-openvswitch\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.634380 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.631941    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-sysctl-conf\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.634380 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.632131    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1bd54f8-715f-428f-998b-65c6e7f8830e-cni-binary-copy\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.634380 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.632166    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9zml\" (UniqueName: \"kubernetes.io/projected/18dcacc0-751d-4f2e-8101-59a1da48fab2-kube-api-access-g9zml\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.634380 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.632196    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-conf-dir\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.634380 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.632269    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-conf-dir\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.634380 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.632333    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1bd54f8-715f-428f-998b-65c6e7f8830e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.634380 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.633086    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-ovnkube-config\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.634380 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.633261    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1bd54f8-715f-428f-998b-65c6e7f8830e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.634380 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.633270    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-ovn-node-metrics-cert\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.634380 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.633397    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0dfa862-aaf8-4ff1-820e-a610a4d41ba8-agent-certs\") pod \"konnectivity-agent-6zx4w\" (UID: \"a0dfa862-aaf8-4ff1-820e-a610a4d41ba8\") " pod="kube-system/konnectivity-agent-6zx4w"
Nov 02 22:52:02.634380 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.633609    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18dcacc0-751d-4f2e-8101-59a1da48fab2-tmp\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.634817 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.633862    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-tuned\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.681619 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:02.681563    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 22:52:02.681619 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:02.681585    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 22:52:02.681619 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:02.681595    2582 projected.go:196] Error preparing data for projected volume kube-api-access-drxcv for pod openshift-network-diagnostics/network-check-target-qhbsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 22:52:02.681825 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:02.681638    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv podName:8161cb48-56bb-455b-bee6-cd3d06607109 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:03.181626813 +0000 UTC m=+2.294394806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-drxcv" (UniqueName: "kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv") pod "network-check-target-qhbsn" (UID: "8161cb48-56bb-455b-bee6-cd3d06607109") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 22:52:02.684253 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.684101    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxxqd\" (UniqueName: \"kubernetes.io/projected/11efd367-1b70-4cb5-93c5-2dda7b9bb010-kube-api-access-nxxqd\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.684759 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.684491    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9qmn\" (UniqueName: \"kubernetes.io/projected/e37c6a5d-1170-4f2d-ae18-2580da36f53b-kube-api-access-j9qmn\") pod \"iptables-alerter-p5hzg\" (UID: \"e37c6a5d-1170-4f2d-ae18-2580da36f53b\") " pod="openshift-network-operator/iptables-alerter-p5hzg"
Nov 02 22:52:02.684759 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.684499    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25fvj\" (UniqueName: \"kubernetes.io/projected/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-kube-api-access-25fvj\") pod \"network-metrics-daemon-6gn7z\" (UID: \"4b1cf92b-28eb-4894-96ed-b7eba4a781d2\") " pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:02.684759 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.684670    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4nhs\" (UniqueName: \"kubernetes.io/projected/92845990-417f-4c8f-aa67-ea9961516288-kube-api-access-c4nhs\") pod \"node-resolver-r2vkn\" (UID: \"92845990-417f-4c8f-aa67-ea9961516288\") " pod="openshift-dns/node-resolver-r2vkn"
Nov 02 22:52:02.684759 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.684689    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9m47\" (UniqueName: \"kubernetes.io/projected/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e-kube-api-access-x9m47\") pod \"node-ca-hfl65\" (UID: \"8e71e09c-79f2-42d0-a2ee-e1535cf3df8e\") " pod="openshift-image-registry/node-ca-hfl65"
Nov 02 22:52:02.685130 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.685112    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hl8b\" (UniqueName: \"kubernetes.io/projected/b1bd54f8-715f-428f-998b-65c6e7f8830e-kube-api-access-5hl8b\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.686095 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.686071    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-456qc\" (UniqueName: \"kubernetes.io/projected/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-kube-api-access-456qc\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.686599 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.686578    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6hww\" (UniqueName: \"kubernetes.io/projected/0ce80856-f167-427e-b8d5-4b856bfcf1ea-kube-api-access-r6hww\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.686992 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.686956    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9zml\" (UniqueName: \"kubernetes.io/projected/18dcacc0-751d-4f2e-8101-59a1da48fab2-kube-api-access-g9zml\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.789158 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.789123    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:02.792243 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.792216    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6zx4w"
Nov 02 22:52:02.820631 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.820610    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:02.851397 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.851368    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r2vkn"
Nov 02 22:52:02.851483 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.851405    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:02.854463 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.854436    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:02.869882 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.869866    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p5hzg"
Nov 02 22:52:02.886666 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.886644    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:02.888773 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:02.888757    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hfl65"
Nov 02 22:52:03.036594 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:52:03.036557    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda401e1afc8eeebc2962b0eea90d39ec6.slice/crio-87fe95f3eb83273324f4440483aac39cb7cf35a79c053349b628fb609ed9c00d WatchSource:0}: Error finding container 87fe95f3eb83273324f4440483aac39cb7cf35a79c053349b628fb609ed9c00d: Status 404 returned error can't find the container with id 87fe95f3eb83273324f4440483aac39cb7cf35a79c053349b628fb609ed9c00d
Nov 02 22:52:03.037111 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:52:03.037090    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990ba2ba1d2d380f95d805ba47d10a19.slice/crio-2dcd50348bcf9c60f8141439ad5bc51f5e57c17db613473548270e0ebaa46be2 WatchSource:0}: Error finding container 2dcd50348bcf9c60f8141439ad5bc51f5e57c17db613473548270e0ebaa46be2: Status 404 returned error can't find the container with id 2dcd50348bcf9c60f8141439ad5bc51f5e57c17db613473548270e0ebaa46be2
Nov 02 22:52:03.042429 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:03.042411    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 22:52:03.136658 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:03.136620    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:03.136828 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:03.136684    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs\") pod \"network-metrics-daemon-6gn7z\" (UID: \"4b1cf92b-28eb-4894-96ed-b7eba4a781d2\") " pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:03.136828 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:03.136780    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 22:52:03.136893 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:03.136837    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret podName:3a474267-46b1-4759-9018-54960f751878 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:04.13682414 +0000 UTC m=+3.249592146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret") pod "global-pull-secret-syncer-hlfzd" (UID: "3a474267-46b1-4759-9018-54960f751878") : object "kube-system"/"original-pull-secret" not registered
Nov 02 22:52:03.136893 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:03.136780    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 22:52:03.136893 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:03.136875    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs podName:4b1cf92b-28eb-4894-96ed-b7eba4a781d2 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:04.136865302 +0000 UTC m=+3.249633310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs") pod "network-metrics-daemon-6gn7z" (UID: "4b1cf92b-28eb-4894-96ed-b7eba4a781d2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 22:52:03.237304 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:03.237266    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxcv\" (UniqueName: \"kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv\") pod \"network-check-target-qhbsn\" (UID: \"8161cb48-56bb-455b-bee6-cd3d06607109\") " pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:03.237831 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:03.237385    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 22:52:03.237831 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:03.237398    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 22:52:03.237831 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:03.237409    2582 projected.go:196] Error preparing data for projected volume kube-api-access-drxcv for pod openshift-network-diagnostics/network-check-target-qhbsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 22:52:03.237831 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:03.237454    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv podName:8161cb48-56bb-455b-bee6-cd3d06607109 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:04.237442165 +0000 UTC m=+3.350210173 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-drxcv" (UniqueName: "kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv") pod "network-check-target-qhbsn" (UID: "8161cb48-56bb-455b-bee6-cd3d06607109") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 22:52:03.361073 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:03.360959    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal" event={"ID":"990ba2ba1d2d380f95d805ba47d10a19","Type":"ContainerStarted","Data":"2dcd50348bcf9c60f8141439ad5bc51f5e57c17db613473548270e0ebaa46be2"}
Nov 02 22:52:03.361926 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:03.361903    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal" event={"ID":"a401e1afc8eeebc2962b0eea90d39ec6","Type":"ContainerStarted","Data":"87fe95f3eb83273324f4440483aac39cb7cf35a79c053349b628fb609ed9c00d"}
Nov 02 22:52:03.556376 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:03.556327    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:47:02 +0000 UTC" deadline="2028-06-20 17:03:27.973320777 +0000 UTC"
Nov 02 22:52:03.556376 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:03.556365    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14298h11m24.416958414s"
Nov 02 22:52:03.560482 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:03.560467    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 22:52:03.751410 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:03.751380    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 22:52:03.751643 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:52:03.751617    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce80856_f167_427e_b8d5_4b856bfcf1ea.slice/crio-ae50de4dcc40e8a529963141abb98c14f84a45dc0414b2cf83fc93745d2d6284 WatchSource:0}: Error finding container ae50de4dcc40e8a529963141abb98c14f84a45dc0414b2cf83fc93745d2d6284: Status 404 returned error can't find the container with id ae50de4dcc40e8a529963141abb98c14f84a45dc0414b2cf83fc93745d2d6284
Nov 02 22:52:03.773545 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:52:03.773522    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18dcacc0_751d_4f2e_8101_59a1da48fab2.slice/crio-6cecc5624ef65086684767381274e174994cb49e6bb774e4c2d22c35034dfe42 WatchSource:0}: Error finding container 6cecc5624ef65086684767381274e174994cb49e6bb774e4c2d22c35034dfe42: Status 404 returned error can't find the container with id 6cecc5624ef65086684767381274e174994cb49e6bb774e4c2d22c35034dfe42
Nov 02 22:52:03.774590 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:52:03.774569    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11efd367_1b70_4cb5_93c5_2dda7b9bb010.slice/crio-9d11b1994899cb37357fb0763c08a84a782e32827d09bd6049b89c1023e3ba60 WatchSource:0}: Error finding container 9d11b1994899cb37357fb0763c08a84a782e32827d09bd6049b89c1023e3ba60: Status 404 returned error can't find the container with id 9d11b1994899cb37357fb0763c08a84a782e32827d09bd6049b89c1023e3ba60
Nov 02 22:52:03.789701 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:52:03.789675    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8c537b_69a3_495b_a99b_afcbdc20ddf6.slice/crio-b5580e7c20fe728be4563a67ca5e2e8a3164967866d28f8231960313f313aa9e WatchSource:0}: Error finding container b5580e7c20fe728be4563a67ca5e2e8a3164967866d28f8231960313f313aa9e: Status 404 returned error can't find the container with id b5580e7c20fe728be4563a67ca5e2e8a3164967866d28f8231960313f313aa9e
Nov 02 22:52:03.822010 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:52:03.821967    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0dfa862_aaf8_4ff1_820e_a610a4d41ba8.slice/crio-a7e7a4e327a441143ba1a60932f1b143231e7f12758ffaa6134e82b54ad4b467 WatchSource:0}: Error finding container a7e7a4e327a441143ba1a60932f1b143231e7f12758ffaa6134e82b54ad4b467: Status 404 returned error can't find the container with id a7e7a4e327a441143ba1a60932f1b143231e7f12758ffaa6134e82b54ad4b467
Nov 02 22:52:03.826140 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:52:03.826096    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e71e09c_79f2_42d0_a2ee_e1535cf3df8e.slice/crio-943ab21fc7bc5cc5287fafd80dd31f5df3efa877190b6a6cbdb8d94ba508d2b3 WatchSource:0}: Error finding container 943ab21fc7bc5cc5287fafd80dd31f5df3efa877190b6a6cbdb8d94ba508d2b3: Status 404 returned error can't find the container with id 943ab21fc7bc5cc5287fafd80dd31f5df3efa877190b6a6cbdb8d94ba508d2b3
Nov 02 22:52:03.828461 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:52:03.828435    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1bd54f8_715f_428f_998b_65c6e7f8830e.slice/crio-a15ab56b9280ee70cea0943cf2842d69055df959b7060863e107f4fd75e0c7b2 WatchSource:0}: Error finding container a15ab56b9280ee70cea0943cf2842d69055df959b7060863e107f4fd75e0c7b2: Status 404 returned error can't find the container with id a15ab56b9280ee70cea0943cf2842d69055df959b7060863e107f4fd75e0c7b2
Nov 02 22:52:03.829145 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:52:03.829124    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92845990_417f_4c8f_aa67_ea9961516288.slice/crio-91450f8b91b9d1ab9ef1e553b441595f4093d010afd7144301deb6db46924170 WatchSource:0}: Error finding container 91450f8b91b9d1ab9ef1e553b441595f4093d010afd7144301deb6db46924170: Status 404 returned error can't find the container with id 91450f8b91b9d1ab9ef1e553b441595f4093d010afd7144301deb6db46924170
Nov 02 22:52:03.935220 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:52:03.935193    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37c6a5d_1170_4f2d_ae18_2580da36f53b.slice/crio-cbd4a1fd00f652af41d91fc4b9d669d983a3974c6c5f1509a11389c197b04e23 WatchSource:0}: Error finding container cbd4a1fd00f652af41d91fc4b9d669d983a3974c6c5f1509a11389c197b04e23: Status 404 returned error can't find the container with id cbd4a1fd00f652af41d91fc4b9d669d983a3974c6c5f1509a11389c197b04e23
Nov 02 22:52:04.143372 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:04.143083    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs\") pod \"network-metrics-daemon-6gn7z\" (UID: \"4b1cf92b-28eb-4894-96ed-b7eba4a781d2\") " pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:04.143372 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:04.143155    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:04.143372 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:04.143295    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 22:52:04.143372 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:04.143318    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 22:52:04.144107 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:04.144059    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs podName:4b1cf92b-28eb-4894-96ed-b7eba4a781d2 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:06.143353858 +0000 UTC m=+5.256121866 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs") pod "network-metrics-daemon-6gn7z" (UID: "4b1cf92b-28eb-4894-96ed-b7eba4a781d2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 22:52:04.144226 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:04.144182    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret podName:3a474267-46b1-4759-9018-54960f751878 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:06.144106329 +0000 UTC m=+5.256874341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret") pod "global-pull-secret-syncer-hlfzd" (UID: "3a474267-46b1-4759-9018-54960f751878") : object "kube-system"/"original-pull-secret" not registered
Nov 02 22:52:04.244126 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:04.244045    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxcv\" (UniqueName: \"kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv\") pod \"network-check-target-qhbsn\" (UID: \"8161cb48-56bb-455b-bee6-cd3d06607109\") " pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:04.244740 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:04.244247    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 22:52:04.244740 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:04.244267    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 22:52:04.244740 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:04.244276    2582 projected.go:196] Error preparing data for projected volume kube-api-access-drxcv for pod openshift-network-diagnostics/network-check-target-qhbsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 22:52:04.244740 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:04.244330    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv podName:8161cb48-56bb-455b-bee6-cd3d06607109 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:06.244312638 +0000 UTC m=+5.357080635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-drxcv" (UniqueName: "kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv") pod "network-check-target-qhbsn" (UID: "8161cb48-56bb-455b-bee6-cd3d06607109") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 22:52:04.357258 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:04.357154    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:04.357258 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:04.357198    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:04.357258 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:04.357198    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:04.357538 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:04.357293    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhbsn" podUID="8161cb48-56bb-455b-bee6-cd3d06607109"
Nov 02 22:52:04.357538 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:04.357393    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hlfzd" podUID="3a474267-46b1-4759-9018-54960f751878"
Nov 02 22:52:04.357538 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:04.357446    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gn7z" podUID="4b1cf92b-28eb-4894-96ed-b7eba4a781d2"
Nov 02 22:52:04.364995 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:04.364958    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbhxn" event={"ID":"b1bd54f8-715f-428f-998b-65c6e7f8830e","Type":"ContainerStarted","Data":"a15ab56b9280ee70cea0943cf2842d69055df959b7060863e107f4fd75e0c7b2"}
Nov 02 22:52:04.366618 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:04.366589    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-295c4" event={"ID":"5b8c537b-69a3-495b-a99b-afcbdc20ddf6","Type":"ContainerStarted","Data":"b5580e7c20fe728be4563a67ca5e2e8a3164967866d28f8231960313f313aa9e"}
Nov 02 22:52:04.368625 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:04.368597    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl" event={"ID":"11efd367-1b70-4cb5-93c5-2dda7b9bb010","Type":"ContainerStarted","Data":"9d11b1994899cb37357fb0763c08a84a782e32827d09bd6049b89c1023e3ba60"}
Nov 02 22:52:04.369894 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:04.369870    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q5cp2" event={"ID":"0ce80856-f167-427e-b8d5-4b856bfcf1ea","Type":"ContainerStarted","Data":"ae50de4dcc40e8a529963141abb98c14f84a45dc0414b2cf83fc93745d2d6284"}
Nov 02 22:52:04.371639 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:04.371616    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p5hzg" event={"ID":"e37c6a5d-1170-4f2d-ae18-2580da36f53b","Type":"ContainerStarted","Data":"cbd4a1fd00f652af41d91fc4b9d669d983a3974c6c5f1509a11389c197b04e23"}
Nov 02 22:52:04.373119 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:04.373096    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r2vkn" event={"ID":"92845990-417f-4c8f-aa67-ea9961516288","Type":"ContainerStarted","Data":"91450f8b91b9d1ab9ef1e553b441595f4093d010afd7144301deb6db46924170"}
Nov 02 22:52:04.374445 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:04.374419    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hfl65" event={"ID":"8e71e09c-79f2-42d0-a2ee-e1535cf3df8e","Type":"ContainerStarted","Data":"943ab21fc7bc5cc5287fafd80dd31f5df3efa877190b6a6cbdb8d94ba508d2b3"}
Nov 02 22:52:04.375800 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:04.375773    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6zx4w" event={"ID":"a0dfa862-aaf8-4ff1-820e-a610a4d41ba8","Type":"ContainerStarted","Data":"a7e7a4e327a441143ba1a60932f1b143231e7f12758ffaa6134e82b54ad4b467"}
Nov 02 22:52:04.377114 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:04.377088    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-smds5" event={"ID":"18dcacc0-751d-4f2e-8101-59a1da48fab2","Type":"ContainerStarted","Data":"6cecc5624ef65086684767381274e174994cb49e6bb774e4c2d22c35034dfe42"}
Nov 02 22:52:04.556735 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:04.556617    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:47:02 +0000 UTC" deadline="2028-05-23 05:35:54.884116135 +0000 UTC"
Nov 02 22:52:04.556735 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:04.556681    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13614h43m50.32743931s"
Nov 02 22:52:06.160396 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:06.160366    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs\") pod \"network-metrics-daemon-6gn7z\" (UID: \"4b1cf92b-28eb-4894-96ed-b7eba4a781d2\") " pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:06.160968 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:06.160432    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:06.160968 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:06.160559    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 22:52:06.160968 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:06.160622    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret podName:3a474267-46b1-4759-9018-54960f751878 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:10.160603723 +0000 UTC m=+9.273371731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret") pod "global-pull-secret-syncer-hlfzd" (UID: "3a474267-46b1-4759-9018-54960f751878") : object "kube-system"/"original-pull-secret" not registered
Nov 02 22:52:06.160968 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:06.160686    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 22:52:06.160968 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:06.160718    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs podName:4b1cf92b-28eb-4894-96ed-b7eba4a781d2 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:10.160707211 +0000 UTC m=+9.273475217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs") pod "network-metrics-daemon-6gn7z" (UID: "4b1cf92b-28eb-4894-96ed-b7eba4a781d2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 22:52:06.263290 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:06.263239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxcv\" (UniqueName: \"kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv\") pod \"network-check-target-qhbsn\" (UID: \"8161cb48-56bb-455b-bee6-cd3d06607109\") " pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:06.263484 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:06.263463    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 22:52:06.263565 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:06.263496    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 22:52:06.263565 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:06.263508    2582 projected.go:196] Error preparing data for projected volume kube-api-access-drxcv for pod openshift-network-diagnostics/network-check-target-qhbsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 22:52:06.263681 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:06.263570    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv podName:8161cb48-56bb-455b-bee6-cd3d06607109 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:10.263551497 +0000 UTC m=+9.376319507 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-drxcv" (UniqueName: "kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv") pod "network-check-target-qhbsn" (UID: "8161cb48-56bb-455b-bee6-cd3d06607109") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 22:52:06.358133 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:06.357923    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:06.358271 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:06.358246    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hlfzd" podUID="3a474267-46b1-4759-9018-54960f751878"
Nov 02 22:52:06.358321 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:06.358108    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:06.358374 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:06.358359    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gn7z" podUID="4b1cf92b-28eb-4894-96ed-b7eba4a781d2"
Nov 02 22:52:06.358374 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:06.358059    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:06.358473 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:06.358434    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhbsn" podUID="8161cb48-56bb-455b-bee6-cd3d06607109"
Nov 02 22:52:06.406161 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:06.406120    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal" event={"ID":"990ba2ba1d2d380f95d805ba47d10a19","Type":"ContainerStarted","Data":"9a0e34bc268dc13e29f9b726d2535f600c650d683f9221bbdb4a6a22be2df0dc"}
Nov 02 22:52:06.409586 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:06.409554    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal" event={"ID":"a401e1afc8eeebc2962b0eea90d39ec6","Type":"ContainerStarted","Data":"200213f77723760ed486da6b92f7370c9a6272689532a2e44bfc4f4e2ee2bd3c"}
Nov 02 22:52:06.621115 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:06.620968    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal" podStartSLOduration=5.62094621 podStartE2EDuration="5.62094621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:06.513726516 +0000 UTC m=+5.626494543" watchObservedRunningTime="2026-11-02 22:52:06.62094621 +0000 UTC m=+5.733714228"
Nov 02 22:52:07.429410 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:07.429382    2582 generic.go:356] "Generic (PLEG): container finished" podID="a401e1afc8eeebc2962b0eea90d39ec6" containerID="200213f77723760ed486da6b92f7370c9a6272689532a2e44bfc4f4e2ee2bd3c" exitCode=0
Nov 02 22:52:07.437440 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:07.434613    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal" event={"ID":"a401e1afc8eeebc2962b0eea90d39ec6","Type":"ContainerDied","Data":"200213f77723760ed486da6b92f7370c9a6272689532a2e44bfc4f4e2ee2bd3c"}
Nov 02 22:52:08.358553 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:08.358040    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:08.358553 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:08.358184    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gn7z" podUID="4b1cf92b-28eb-4894-96ed-b7eba4a781d2"
Nov 02 22:52:08.358553 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:08.358303    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:08.358553 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:08.358380    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hlfzd" podUID="3a474267-46b1-4759-9018-54960f751878"
Nov 02 22:52:08.358553 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:08.358435    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:08.358553 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:08.358500    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhbsn" podUID="8161cb48-56bb-455b-bee6-cd3d06607109"
Nov 02 22:52:08.441100 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:08.441061    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal" event={"ID":"a401e1afc8eeebc2962b0eea90d39ec6","Type":"ContainerStarted","Data":"983c1dbdad59fb51d40e5ce48b0cbbfeae86ac7724b92ecd31056b1d1bcdf682"}
Nov 02 22:52:08.452587 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:08.451858    2582 generic.go:356] "Generic (PLEG): container finished" podID="b1bd54f8-715f-428f-998b-65c6e7f8830e" containerID="e508006d6bd0317f5109bc1293f57bf6883d22bad6b14486b7e69ca7e0f03648" exitCode=0
Nov 02 22:52:08.452587 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:08.451913    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbhxn" event={"ID":"b1bd54f8-715f-428f-998b-65c6e7f8830e","Type":"ContainerDied","Data":"e508006d6bd0317f5109bc1293f57bf6883d22bad6b14486b7e69ca7e0f03648"}
Nov 02 22:52:08.660052 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:08.659935    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal" podStartSLOduration=7.659918368 podStartE2EDuration="7.659918368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:08.549296584 +0000 UTC m=+7.662064601" watchObservedRunningTime="2026-11-02 22:52:08.659918368 +0000 UTC m=+7.772686385"
Nov 02 22:52:09.459164 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:09.459026    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6zx4w" event={"ID":"a0dfa862-aaf8-4ff1-820e-a610a4d41ba8","Type":"ContainerStarted","Data":"a92985df2664e240edf3744af1ddd01ef51b72652d1ebe475155f15c1811e464"}
Nov 02 22:52:10.196558 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:10.195804    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs\") pod \"network-metrics-daemon-6gn7z\" (UID: \"4b1cf92b-28eb-4894-96ed-b7eba4a781d2\") " pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:10.196558 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:10.195880    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:10.196558 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:10.196032    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 22:52:10.196558 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:10.196100    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret podName:3a474267-46b1-4759-9018-54960f751878 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:18.196081344 +0000 UTC m=+17.308849361 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret") pod "global-pull-secret-syncer-hlfzd" (UID: "3a474267-46b1-4759-9018-54960f751878") : object "kube-system"/"original-pull-secret" not registered
Nov 02 22:52:10.196558 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:10.196490    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 22:52:10.196558 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:10.196533    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs podName:4b1cf92b-28eb-4894-96ed-b7eba4a781d2 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:18.196518749 +0000 UTC m=+17.309286746 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs") pod "network-metrics-daemon-6gn7z" (UID: "4b1cf92b-28eb-4894-96ed-b7eba4a781d2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 22:52:10.296806 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:10.296430    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxcv\" (UniqueName: \"kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv\") pod \"network-check-target-qhbsn\" (UID: \"8161cb48-56bb-455b-bee6-cd3d06607109\") " pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:10.296806 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:10.296621    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 22:52:10.296806 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:10.296645    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 22:52:10.296806 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:10.296658    2582 projected.go:196] Error preparing data for projected volume kube-api-access-drxcv for pod openshift-network-diagnostics/network-check-target-qhbsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 22:52:10.296806 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:10.296718    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv podName:8161cb48-56bb-455b-bee6-cd3d06607109 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:18.296699815 +0000 UTC m=+17.409467828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-drxcv" (UniqueName: "kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv") pod "network-check-target-qhbsn" (UID: "8161cb48-56bb-455b-bee6-cd3d06607109") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 22:52:10.358424 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:10.357585    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:10.358424 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:10.357727    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hlfzd" podUID="3a474267-46b1-4759-9018-54960f751878"
Nov 02 22:52:10.358424 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:10.358102    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:10.358424 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:10.358191    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhbsn" podUID="8161cb48-56bb-455b-bee6-cd3d06607109"
Nov 02 22:52:10.358424 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:10.358256    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:10.358424 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:10.358347    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gn7z" podUID="4b1cf92b-28eb-4894-96ed-b7eba4a781d2"
Nov 02 22:52:10.464736 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:10.464072    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hfl65" event={"ID":"8e71e09c-79f2-42d0-a2ee-e1535cf3df8e","Type":"ContainerStarted","Data":"01bf9de8840d88e9cab3d602ca36ee150f9dc07b74716644b5a7cdc358855c1a"}
Nov 02 22:52:10.467452 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:10.467340    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl" event={"ID":"11efd367-1b70-4cb5-93c5-2dda7b9bb010","Type":"ContainerStarted","Data":"9416435303e3c1d0a62ae4c93355032d17463da0a8e7984ae3156da0a69267e5"}
Nov 02 22:52:10.573034 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:10.571905    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6zx4w" podStartSLOduration=5.171268222 podStartE2EDuration="9.571884565s" totalImagesPullingTime="4.400616343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:09.567384387 +0000 UTC m=+8.680152403" watchObservedRunningTime="2026-11-02 22:52:10.571884565 +0000 UTC m=+9.684652584"
Nov 02 22:52:12.242693 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:12.242663    2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 22:52:12.277321 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:12.277200    2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T22:52:12.242689516Z","UUID":"7a7908c2-dc56-4068-9ae1-1617bc6f33c2","Handler":null,"Name":"","Endpoint":""}
Nov 02 22:52:12.278848 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:12.278812    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 22:52:12.278848 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:12.278843    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 22:52:12.357581 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:12.357549    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:12.357756 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:12.357658    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhbsn" podUID="8161cb48-56bb-455b-bee6-cd3d06607109"
Nov 02 22:52:12.357756 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:12.357723    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:12.357859 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:12.357846    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gn7z" podUID="4b1cf92b-28eb-4894-96ed-b7eba4a781d2"
Nov 02 22:52:12.357891 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:12.357878    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:12.357953 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:12.357937    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hlfzd" podUID="3a474267-46b1-4759-9018-54960f751878"
Nov 02 22:52:12.479817 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:12.479708    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl" event={"ID":"11efd367-1b70-4cb5-93c5-2dda7b9bb010","Type":"ContainerStarted","Data":"88bba8a516749a8d18a57de08b34a42de47c928b484497d9390fa78fed2c76bb"}
Nov 02 22:52:12.481948 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:12.481909    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q5cp2" event={"ID":"0ce80856-f167-427e-b8d5-4b856bfcf1ea","Type":"ContainerStarted","Data":"81288313c8dbb8f4d43a7da7a4063d9a0e183ba907c0afd6f1eb257492d287a2"}
Nov 02 22:52:12.597956 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:12.597900    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hfl65" podStartSLOduration=5.728551524 podStartE2EDuration="11.597886037s" totalImagesPullingTime="5.869334513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:10.572927499 +0000 UTC m=+9.685695516" watchObservedRunningTime="2026-11-02 22:52:12.597886037 +0000 UTC m=+11.710654052"
Nov 02 22:52:13.487566 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:13.487356    2582 generic.go:356] "Generic (PLEG): container finished" podID="b1bd54f8-715f-428f-998b-65c6e7f8830e" containerID="f34c6059a516d1cca0f150efe569ef21f4ecfbaac50fbdf29d5128008365612d" exitCode=0
Nov 02 22:52:13.487566 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:13.487549    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbhxn" event={"ID":"b1bd54f8-715f-428f-998b-65c6e7f8830e","Type":"ContainerDied","Data":"f34c6059a516d1cca0f150efe569ef21f4ecfbaac50fbdf29d5128008365612d"}
Nov 02 22:52:13.489802 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:13.489783    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r2vkn" event={"ID":"92845990-417f-4c8f-aa67-ea9961516288","Type":"ContainerStarted","Data":"4940f82b7491f34fa7b7ff20031fe0eb250effd1d75f4c121d5725f47c520cb6"}
Nov 02 22:52:13.694501 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:13.694400    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-q5cp2" podStartSLOduration=4.543383437 podStartE2EDuration="12.69438056s" totalImagesPullingTime="8.150997123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:12.598649144 +0000 UTC m=+11.711417166" watchObservedRunningTime="2026-11-02 22:52:13.69438056 +0000 UTC m=+12.807148577"
Nov 02 22:52:13.802356 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:13.802304    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-r2vkn" podStartSLOduration=4.220350176 podStartE2EDuration="12.80228351s" totalImagesPullingTime="8.581933334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:13.801690302 +0000 UTC m=+12.914458323" watchObservedRunningTime="2026-11-02 22:52:13.80228351 +0000 UTC m=+12.915051527"
Nov 02 22:52:14.357207 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:14.357174    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:14.357391 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:14.357174    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:14.357391 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:14.357292    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gn7z" podUID="4b1cf92b-28eb-4894-96ed-b7eba4a781d2"
Nov 02 22:52:14.357391 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:14.357339    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:14.357535 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:14.357440    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhbsn" podUID="8161cb48-56bb-455b-bee6-cd3d06607109"
Nov 02 22:52:14.357591 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:14.357525    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hlfzd" podUID="3a474267-46b1-4759-9018-54960f751878"
Nov 02 22:52:14.370309 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:14.370275    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6zx4w"
Nov 02 22:52:14.371053 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:14.371027    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6zx4w"
Nov 02 22:52:14.494125 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:14.494086    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl" event={"ID":"11efd367-1b70-4cb5-93c5-2dda7b9bb010","Type":"ContainerStarted","Data":"ae05f4d94a5aa0d3d141470cda4da8415bb581cff2ec5f70f500a6910df38618"}
Nov 02 22:52:14.496186 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:14.496157    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p5hzg" event={"ID":"e37c6a5d-1170-4f2d-ae18-2580da36f53b","Type":"ContainerStarted","Data":"e96bafa296455cd933d225851ce975ea4b497bcdfbeea96782a55ec581c39f2c"}
Nov 02 22:52:14.605844 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:14.605787    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl" podStartSLOduration=4.013401106 podStartE2EDuration="13.605768827s" totalImagesPullingTime="9.592367721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:14.605332493 +0000 UTC m=+13.718100510" watchObservedRunningTime="2026-11-02 22:52:14.605768827 +0000 UTC m=+13.718536845"
Nov 02 22:52:14.713243 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:14.713161    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-p5hzg" podStartSLOduration=5.144157944 podStartE2EDuration="13.713147568s" totalImagesPullingTime="8.568989624s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:14.712838718 +0000 UTC m=+13.825606745" watchObservedRunningTime="2026-11-02 22:52:14.713147568 +0000 UTC m=+13.825915585"
Nov 02 22:52:15.500314 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:15.500261    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-smds5" event={"ID":"18dcacc0-751d-4f2e-8101-59a1da48fab2","Type":"ContainerStarted","Data":"256c26762d843adcd5e897e1551425956ebf2fdd5f35bf3a1fece838fb29e1ff"}
Nov 02 22:52:15.503205 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:15.503173    2582 generic.go:356] "Generic (PLEG): container finished" podID="b1bd54f8-715f-428f-998b-65c6e7f8830e" containerID="2068d7388dc78cd4cfe2998de11538078bbc2abb7c535d2fd0ea01120bd12f44" exitCode=0
Nov 02 22:52:15.503528 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:15.503501    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbhxn" event={"ID":"b1bd54f8-715f-428f-998b-65c6e7f8830e","Type":"ContainerDied","Data":"2068d7388dc78cd4cfe2998de11538078bbc2abb7c535d2fd0ea01120bd12f44"}
Nov 02 22:52:15.610090 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:15.610041    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-smds5" podStartSLOduration=3.94353882 podStartE2EDuration="14.610026483s" totalImagesPullingTime="10.666487663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:15.60978616 +0000 UTC m=+14.722554233" watchObservedRunningTime="2026-11-02 22:52:15.610026483 +0000 UTC m=+14.722794495"
Nov 02 22:52:16.357895 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:16.357858    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:16.358061 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:16.357921    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:16.358061 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:16.358034    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:16.358061 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:16.358043    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hlfzd" podUID="3a474267-46b1-4759-9018-54960f751878"
Nov 02 22:52:16.358175 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:16.358126    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhbsn" podUID="8161cb48-56bb-455b-bee6-cd3d06607109"
Nov 02 22:52:16.358208 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:16.358190    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gn7z" podUID="4b1cf92b-28eb-4894-96ed-b7eba4a781d2"
Nov 02 22:52:17.511702 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:17.511420    2582 generic.go:356] "Generic (PLEG): container finished" podID="b1bd54f8-715f-428f-998b-65c6e7f8830e" containerID="e90e6debd6f3424cf9933d059a3ee9039bcf83355e74fa8fa3e6835dfe36bf49" exitCode=0
Nov 02 22:52:17.511702 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:17.511504    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbhxn" event={"ID":"b1bd54f8-715f-428f-998b-65c6e7f8830e","Type":"ContainerDied","Data":"e90e6debd6f3424cf9933d059a3ee9039bcf83355e74fa8fa3e6835dfe36bf49"}
Nov 02 22:52:18.263291 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:18.263258    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:18.263459 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:18.263315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs\") pod \"network-metrics-daemon-6gn7z\" (UID: \"4b1cf92b-28eb-4894-96ed-b7eba4a781d2\") " pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:18.263459 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:18.263416    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 22:52:18.263525 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:18.263481    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret podName:3a474267-46b1-4759-9018-54960f751878 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:34.263467487 +0000 UTC m=+33.376235497 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret") pod "global-pull-secret-syncer-hlfzd" (UID: "3a474267-46b1-4759-9018-54960f751878") : object "kube-system"/"original-pull-secret" not registered
Nov 02 22:52:18.263525 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:18.263428    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 22:52:18.263585 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:18.263553    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs podName:4b1cf92b-28eb-4894-96ed-b7eba4a781d2 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:34.263539673 +0000 UTC m=+33.376307681 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs") pod "network-metrics-daemon-6gn7z" (UID: "4b1cf92b-28eb-4894-96ed-b7eba4a781d2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 22:52:18.357898 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:18.357873    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:18.358014 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:18.357899    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:18.358014 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:18.357909    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:18.358014 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:18.358006    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hlfzd" podUID="3a474267-46b1-4759-9018-54960f751878"
Nov 02 22:52:18.358195 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:18.358043    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhbsn" podUID="8161cb48-56bb-455b-bee6-cd3d06607109"
Nov 02 22:52:18.358195 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:18.358139    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gn7z" podUID="4b1cf92b-28eb-4894-96ed-b7eba4a781d2"
Nov 02 22:52:18.364064 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:18.364048    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxcv\" (UniqueName: \"kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv\") pod \"network-check-target-qhbsn\" (UID: \"8161cb48-56bb-455b-bee6-cd3d06607109\") " pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:18.364177 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:18.364165    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 22:52:18.364210 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:18.364180    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 22:52:18.364210 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:18.364188    2582 projected.go:196] Error preparing data for projected volume kube-api-access-drxcv for pod openshift-network-diagnostics/network-check-target-qhbsn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 22:52:18.364279 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:18.364226    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv podName:8161cb48-56bb-455b-bee6-cd3d06607109 nodeName:}" failed. No retries permitted until 2026-11-02 22:52:34.364212431 +0000 UTC m=+33.476980425 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-drxcv" (UniqueName: "kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv") pod "network-check-target-qhbsn" (UID: "8161cb48-56bb-455b-bee6-cd3d06607109") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 22:52:18.516651 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:18.516492    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-295c4" event={"ID":"5b8c537b-69a3-495b-a99b-afcbdc20ddf6","Type":"ContainerStarted","Data":"f1c0acbe25a2010f54699cc63d26793350cf2bfd74bc32bcacf05be7a8b07a94"}
Nov 02 22:52:18.516949 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:18.516697    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-295c4" event={"ID":"5b8c537b-69a3-495b-a99b-afcbdc20ddf6","Type":"ContainerStarted","Data":"b0218ef95041f5fbd59c919d84871c434a9097f20e5a3f33b45ac1551de51fe5"}
Nov 02 22:52:19.521230 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:19.521198    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-295c4" event={"ID":"5b8c537b-69a3-495b-a99b-afcbdc20ddf6","Type":"ContainerStarted","Data":"15e8ff633bf2ddec5a0e17dfa880fe6afcf859d1d2b0317f1b9dc877f9421413"}
Nov 02 22:52:19.521602 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:19.521272    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-295c4" event={"ID":"5b8c537b-69a3-495b-a99b-afcbdc20ddf6","Type":"ContainerStarted","Data":"670bbc3188faf3140c8a93171b359987ab3a9aa8523243a703f6af60369cb2b2"}
Nov 02 22:52:19.521602 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:19.521304    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-295c4" event={"ID":"5b8c537b-69a3-495b-a99b-afcbdc20ddf6","Type":"ContainerStarted","Data":"940ffaec27d74b51c3aef54f7c8baa7a8ffa171b1fb5c8c9e534af4edeaa56a7"}
Nov 02 22:52:19.521602 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:19.521334    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-295c4" event={"ID":"5b8c537b-69a3-495b-a99b-afcbdc20ddf6","Type":"ContainerStarted","Data":"f76ad9aa6530c9924c5f7ffcc0beb890a2321ed35321a7ea5edc1a3c5a5ac4ee"}
Nov 02 22:52:20.358048 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:20.358005    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:20.358241 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:20.358064    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:20.358241 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:20.358072    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:20.358241 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:20.358172    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gn7z" podUID="4b1cf92b-28eb-4894-96ed-b7eba4a781d2"
Nov 02 22:52:20.358393 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:20.358295    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hlfzd" podUID="3a474267-46b1-4759-9018-54960f751878"
Nov 02 22:52:20.358444 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:20.358394    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhbsn" podUID="8161cb48-56bb-455b-bee6-cd3d06607109"
Nov 02 22:52:20.472825 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:20.472784    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:20.526449 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:20.526419    2582 generic.go:356] "Generic (PLEG): container finished" podID="b1bd54f8-715f-428f-998b-65c6e7f8830e" containerID="a9da7d64f1eec3ead3ffe9d980cc570574a5ffdfe9c47a6ec341995417572a8d" exitCode=0
Nov 02 22:52:20.526967 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:20.526471    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbhxn" event={"ID":"b1bd54f8-715f-428f-998b-65c6e7f8830e","Type":"ContainerDied","Data":"a9da7d64f1eec3ead3ffe9d980cc570574a5ffdfe9c47a6ec341995417572a8d"}
Nov 02 22:52:20.685786 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:20.685753    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:21.531416 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:21.531382    2582 generic.go:356] "Generic (PLEG): container finished" podID="b1bd54f8-715f-428f-998b-65c6e7f8830e" containerID="ab9f13bcd95a1f5992b6542e4ee057ca23fe756e89afe5762e0f65685141f7c1" exitCode=0
Nov 02 22:52:21.531922 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:21.531463    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbhxn" event={"ID":"b1bd54f8-715f-428f-998b-65c6e7f8830e","Type":"ContainerDied","Data":"ab9f13bcd95a1f5992b6542e4ee057ca23fe756e89afe5762e0f65685141f7c1"}
Nov 02 22:52:21.534735 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:21.534715    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-295c4" event={"ID":"5b8c537b-69a3-495b-a99b-afcbdc20ddf6","Type":"ContainerStarted","Data":"7aa4cca6cab2fca2a4c268548b850de978e89ada1c108ffc88571049fb53cd6c"}
Nov 02 22:52:22.357216 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:22.357174    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:22.357216 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:22.357196    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:22.357216 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:22.357203    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:22.357465 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:22.357291    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gn7z" podUID="4b1cf92b-28eb-4894-96ed-b7eba4a781d2"
Nov 02 22:52:22.357465 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:22.357348    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhbsn" podUID="8161cb48-56bb-455b-bee6-cd3d06607109"
Nov 02 22:52:22.357465 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:22.357418    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hlfzd" podUID="3a474267-46b1-4759-9018-54960f751878"
Nov 02 22:52:22.539933 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:22.539897    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kbhxn" event={"ID":"b1bd54f8-715f-428f-998b-65c6e7f8830e","Type":"ContainerStarted","Data":"7d44b5d43f53501af9bf4ea20dbd97b84e5f235e98495f29d0169adf8455b538"}
Nov 02 22:52:22.699892 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:22.699792    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kbhxn" podStartSLOduration=9.18556825 podStartE2EDuration="21.699777934s" totalImagesPullingTime="12.514209684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:22.699257357 +0000 UTC m=+21.812025374" watchObservedRunningTime="2026-11-02 22:52:22.699777934 +0000 UTC m=+21.812545950"
Nov 02 22:52:23.058721 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:23.058462    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:23.222851 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:23.222803    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:23.545729 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:23.545659    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-295c4" event={"ID":"5b8c537b-69a3-495b-a99b-afcbdc20ddf6","Type":"ContainerStarted","Data":"614954fcbb9f9c5ce808f586abea206513220786fb1864c33666558798e23672"}
Nov 02 22:52:23.546389 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:23.546364    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:23.561647 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:23.561627    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:23.802457 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:23.802363    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-295c4" podStartSLOduration=8.264816378 podStartE2EDuration="22.802349645s" totalImagesPullingTime="14.537533267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:23.754524972 +0000 UTC m=+22.867292987" watchObservedRunningTime="2026-11-02 22:52:23.802349645 +0000 UTC m=+22.915117661"
Nov 02 22:52:24.288228 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:24.288150    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:24.357845 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:24.357817    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:24.357974 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:24.357849    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:24.357974 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:24.357912    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gn7z" podUID="4b1cf92b-28eb-4894-96ed-b7eba4a781d2"
Nov 02 22:52:24.358078 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:24.358000    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:24.358111 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:24.358081    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hlfzd" podUID="3a474267-46b1-4759-9018-54960f751878"
Nov 02 22:52:24.358155 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:24.358115    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhbsn" podUID="8161cb48-56bb-455b-bee6-cd3d06607109"
Nov 02 22:52:24.445503 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:24.445467    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:24.547651 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:24.547567    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:24.547651 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:24.547650    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:24.562513 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:24.562486    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:24.855040 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:24.854938    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6zx4w"
Nov 02 22:52:24.855183 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:24.855098    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 22:52:24.855505 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:24.855482    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6zx4w"
Nov 02 22:52:25.061301 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:25.061265    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:25.537011 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:25.536907    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6gn7z"]
Nov 02 22:52:25.537179 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:25.537055    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:25.537179 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:25.537171    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gn7z" podUID="4b1cf92b-28eb-4894-96ed-b7eba4a781d2"
Nov 02 22:52:25.537568 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:25.537547    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hlfzd"]
Nov 02 22:52:25.537642 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:25.537621    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:25.537703 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:25.537687    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hlfzd" podUID="3a474267-46b1-4759-9018-54960f751878"
Nov 02 22:52:25.539606 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:25.539583    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qhbsn"]
Nov 02 22:52:25.539688 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:25.539655    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:25.539755 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:25.539740    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhbsn" podUID="8161cb48-56bb-455b-bee6-cd3d06607109"
Nov 02 22:52:27.357810 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:27.357736    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:27.357810 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:27.357754    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:27.358297 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:27.357878    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhbsn" podUID="8161cb48-56bb-455b-bee6-cd3d06607109"
Nov 02 22:52:27.358297 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:27.357920    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gn7z" podUID="4b1cf92b-28eb-4894-96ed-b7eba4a781d2"
Nov 02 22:52:27.358297 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:27.357952    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:27.358297 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:27.358057    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hlfzd" podUID="3a474267-46b1-4759-9018-54960f751878"
Nov 02 22:52:27.800675 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:27.800605    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:27.994412 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:27.994382    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:28.158868 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:28.158789    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:28.319870 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:28.319839    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:29.357551 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:29.357472    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:29.357956 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:29.357473    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:29.357956 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:29.357588    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6gn7z" podUID="4b1cf92b-28eb-4894-96ed-b7eba4a781d2"
Nov 02 22:52:29.357956 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:29.357610    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:29.357956 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:29.357680    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hlfzd" podUID="3a474267-46b1-4759-9018-54960f751878"
Nov 02 22:52:29.357956 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:29.357747    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qhbsn" podUID="8161cb48-56bb-455b-bee6-cd3d06607109"
Nov 02 22:52:29.586498 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:29.586464    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:29.805525 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:29.805446    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:30.190562 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:30.190481    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:30.639792 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:30.639623    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeReady"
Nov 02 22:52:30.640168 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:30.639887    2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 22:52:31.360875 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:31.359539    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:31.360875 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:31.360260    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:31.360875 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:31.360672    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:31.494852 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:31.494817    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 22:52:31.494852 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:31.494809    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 22:52:31.495154 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:31.495137    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 22:52:31.495424 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:31.495408    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5kxql\""
Nov 02 22:52:31.495459 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:31.495414    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2rjj4\""
Nov 02 22:52:31.495459 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:31.495414    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-n2bfg\""
Nov 02 22:52:31.495515 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:31.495467    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 22:52:31.635748 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:31.635670    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:31.963719 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:31.963633    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:31.968077 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:31.968052    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r2vkn_92845990-417f-4c8f-aa67-ea9961516288/dns-node-resolver/0.log"
Nov 02 22:52:32.143386 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:32.143354    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:32.302960 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:32.302890    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:32.488075 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:32.488048    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:32.679550 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:32.679474    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:32.845043 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:32.844973    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:32.968537 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:32.968469    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hfl65_8e71e09c-79f2-42d0-a2ee-e1535cf3df8e/node-ca/0.log"
Nov 02 22:52:33.046063 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:33.046031    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:33.211715 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:33.211683    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:33.417422 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:33.417336    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:33.586407 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:33.586375    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:33.868836 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:33.868800    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:34.073238 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:34.073199    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:34.263732 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:34.263700    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:34.272752 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:34.272728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs\") pod \"network-metrics-daemon-6gn7z\" (UID: \"4b1cf92b-28eb-4894-96ed-b7eba4a781d2\") " pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:34.272826 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:34.272772    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:34.272885 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:34.272865    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 22:52:34.272934 ip-10-0-139-234 kubenswrapper[2582]: E1102 22:52:34.272923    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs podName:4b1cf92b-28eb-4894-96ed-b7eba4a781d2 nodeName:}" failed. No retries permitted until 2026-11-02 22:53:06.272909249 +0000 UTC m=+65.385677243 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs") pod "network-metrics-daemon-6gn7z" (UID: "4b1cf92b-28eb-4894-96ed-b7eba4a781d2") : secret "metrics-daemon-secret" not found
Nov 02 22:52:34.276534 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:34.276511    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:34.373713 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:34.373675    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxcv\" (UniqueName: \"kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv\") pod \"network-check-target-qhbsn\" (UID: \"8161cb48-56bb-455b-bee6-cd3d06607109\") " pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:34.376146 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:34.376127    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drxcv\" (UniqueName: \"kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv\") pod \"network-check-target-qhbsn\" (UID: \"8161cb48-56bb-455b-bee6-cd3d06607109\") " pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:34.377901 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:34.377871    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:34.396302 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:34.396286    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:34.472242 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:34.472199    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:34.665409 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:34.665326    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:34.807349 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:34.807318    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hlfzd"]
Nov 02 22:52:34.810200 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:52:34.810175    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a474267_46b1_4759_9018_54960f751878.slice/crio-8bc543e55ddf52e0658042f60d9155c4ccb6bf8c7497816d696f46aab1f64bea WatchSource:0}: Error finding container 8bc543e55ddf52e0658042f60d9155c4ccb6bf8c7497816d696f46aab1f64bea: Status 404 returned error can't find the container with id 8bc543e55ddf52e0658042f60d9155c4ccb6bf8c7497816d696f46aab1f64bea
Nov 02 22:52:34.854557 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:34.854496    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qhbsn"]
Nov 02 22:52:34.856378 ip-10-0-139-234 kubenswrapper[2582]: W1102 22:52:34.856349    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8161cb48_56bb_455b_bee6_cd3d06607109.slice/crio-95d2e09830c2b2560a1284968047f359c1c9a54041e06a7db62d38e4d73ae8e7 WatchSource:0}: Error finding container 95d2e09830c2b2560a1284968047f359c1c9a54041e06a7db62d38e4d73ae8e7: Status 404 returned error can't find the container with id 95d2e09830c2b2560a1284968047f359c1c9a54041e06a7db62d38e4d73ae8e7
Nov 02 22:52:35.575669 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:35.575625    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hlfzd" event={"ID":"3a474267-46b1-4759-9018-54960f751878","Type":"ContainerStarted","Data":"8bc543e55ddf52e0658042f60d9155c4ccb6bf8c7497816d696f46aab1f64bea"}
Nov 02 22:52:35.576788 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:35.576762    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qhbsn" event={"ID":"8161cb48-56bb-455b-bee6-cd3d06607109","Type":"ContainerStarted","Data":"95d2e09830c2b2560a1284968047f359c1c9a54041e06a7db62d38e4d73ae8e7"}
Nov 02 22:52:35.867186 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:35.867105    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:36.058481 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:36.058447    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:36.269178 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:36.269120    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:36.465567 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:36.465527    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:36.622085 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:36.621955    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:36.787487 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:36.787450    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:37.618356 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:37.618321    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:37.820363 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:37.820322    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:37.976217 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:37.976117    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:38.140710 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:38.140678    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:38.294053 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:38.294014    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:38.478219 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:38.478179    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:38.645931 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:38.645901    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:38.804436 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:38.804357    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:39.513217 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:39.513178    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:39.588080 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:39.588043    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qhbsn" event={"ID":"8161cb48-56bb-455b-bee6-cd3d06607109","Type":"ContainerStarted","Data":"99f1717ed7a4638cb4ca4b6b408ee7a4a3773f23620640aa659478e5448d28f6"}
Nov 02 22:52:39.588271 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:39.588176    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:39.939753 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:39.939671    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:40.093222 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:40.093189    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:40.268741 ip-10-0-139-234 kubenswrapper[2582]: I1102 22:52:40.268683    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:40.382208 ip-10-0-139-234 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 22:52:40.393376 ip-10-0-139-234 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 22:52:40.393542 ip-10-0-139-234 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 22:52:40.393591 ip-10-0-139-234 systemd[1]: kubelet.service: Consumed 2.231s CPU time.
Nov 02 22:52:40.402601 ip-10-0-139-234 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 22:52:40.465302 ip-10-0-139-234 kubenswrapper[4294]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 22:52:40.465302 ip-10-0-139-234 kubenswrapper[4294]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 22:52:40.465302 ip-10-0-139-234 kubenswrapper[4294]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 22:52:40.465302 ip-10-0-139-234 kubenswrapper[4294]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 22:52:40.468083 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.467971    4294 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 22:52:40.468083 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468082    4294 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 22:52:40.468083 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468086    4294 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 22:52:40.468213 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468089    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 22:52:40.468213 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468092    4294 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 22:52:40.468213 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468095    4294 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 22:52:40.468213 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468098    4294 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 22:52:40.468213 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468101    4294 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 22:52:40.468213 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468104    4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 22:52:40.468213 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468107    4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 22:52:40.468213 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468110    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 22:52:40.468213 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468113    4294 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 22:52:40.468213 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468116    4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 22:52:40.468213 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468118    4294 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 22:52:40.468213 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468121    4294 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 22:52:40.468213 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468124    4294 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 22:52:40.468213 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468127    4294 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 22:52:40.468213 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468129    4294 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 22:52:40.468213 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468132    4294 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 22:52:40.468213 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468134    4294 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 22:52:40.468213 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468137    4294 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 22:52:40.468213 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468139    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 22:52:40.468213 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468142    4294 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 22:52:40.468711 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468145    4294 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 22:52:40.468711 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468148    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 22:52:40.468711 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468150    4294 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 22:52:40.468711 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468153    4294 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 22:52:40.468711 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468156    4294 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 22:52:40.468711 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468167    4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 22:52:40.468711 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468170    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 22:52:40.468711 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468172    4294 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 22:52:40.468711 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468175    4294 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 22:52:40.468711 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468178    4294 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 22:52:40.468711 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468180    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 22:52:40.468711 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468183    4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 22:52:40.468711 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468186    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 22:52:40.468711 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468189    4294 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 22:52:40.468711 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468192    4294 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 22:52:40.468711 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468195    4294 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 22:52:40.468711 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468197    4294 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 22:52:40.468711 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468200    4294 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 22:52:40.468711 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468203    4294 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 22:52:40.468711 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468206    4294 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 22:52:40.469229 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468208    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 22:52:40.469229 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468211    4294 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 22:52:40.469229 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468213    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 22:52:40.469229 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468216    4294 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 22:52:40.469229 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468219    4294 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 22:52:40.469229 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468223    4294 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 22:52:40.469229 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468226    4294 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 22:52:40.469229 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468229    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 22:52:40.469229 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468231    4294 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 22:52:40.469229 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468234    4294 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 22:52:40.469229 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468236    4294 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 22:52:40.469229 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468239    4294 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 22:52:40.469229 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468241    4294 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 22:52:40.469229 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468245    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 22:52:40.469229 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468247    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 22:52:40.469229 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468250    4294 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 22:52:40.469229 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468253    4294 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 22:52:40.469229 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468255    4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 22:52:40.469229 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468258    4294 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 22:52:40.469229 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468262    4294 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 22:52:40.469730 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468264    4294 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 22:52:40.469730 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468267    4294 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 22:52:40.469730 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468269    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 22:52:40.469730 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468272    4294 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 22:52:40.469730 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468275    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 22:52:40.469730 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468278    4294 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 22:52:40.469730 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468281    4294 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 22:52:40.469730 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468284    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 22:52:40.469730 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468286    4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 22:52:40.469730 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468289    4294 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 22:52:40.469730 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468291    4294 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 22:52:40.469730 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468294    4294 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 22:52:40.469730 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468296    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 22:52:40.469730 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468301    4294 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 22:52:40.469730 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468304    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 22:52:40.469730 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468307    4294 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 22:52:40.469730 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468310    4294 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 22:52:40.469730 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468313    4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 22:52:40.469730 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468315    4294 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 22:52:40.470221 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468318    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 22:52:40.470221 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468321    4294 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 22:52:40.470221 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468323    4294 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 22:52:40.470221 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468326    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 22:52:40.470221 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468330    4294 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 22:52:40.470221 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468350    4294 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 22:52:40.470221 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468353    4294 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 22:52:40.470221 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468357    4294 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 22:52:40.470221 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468359    4294 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 22:52:40.470221 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468363    4294 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 22:52:40.470221 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468365    4294 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 22:52:40.470221 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468368    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 22:52:40.470221 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468371    4294 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 22:52:40.470221 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468374    4294 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 22:52:40.470221 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468378    4294 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 22:52:40.470221 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468382    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 22:52:40.470221 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468384    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 22:52:40.470221 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468387    4294 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 22:52:40.470221 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468390    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 22:52:40.470723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468393    4294 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 22:52:40.470723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468395    4294 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 22:52:40.470723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468398    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 22:52:40.470723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468401    4294 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 22:52:40.470723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.468403    4294 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 22:52:40.470723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469172    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 22:52:40.470723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469179    4294 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 22:52:40.470723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469182    4294 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 22:52:40.470723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469185    4294 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 22:52:40.470723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469189    4294 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 22:52:40.470723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469192    4294 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 22:52:40.470723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469194    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 22:52:40.470723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469197    4294 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 22:52:40.470723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469200    4294 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 22:52:40.470723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469203    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 22:52:40.470723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469206    4294 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 22:52:40.470723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469209    4294 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 22:52:40.470723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469211    4294 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 22:52:40.470723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469214    4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 22:52:40.470723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469216    4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 22:52:40.471243 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469219    4294 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 22:52:40.471243 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469222    4294 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 22:52:40.471243 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469224    4294 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 22:52:40.471243 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469227    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 22:52:40.471243 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469230    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 22:52:40.471243 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469233    4294 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 22:52:40.471243 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469236    4294 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 22:52:40.471243 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469241    4294 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 22:52:40.471243 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469245    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 22:52:40.471243 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469247    4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 22:52:40.471243 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469250    4294 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 22:52:40.471243 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469253    4294 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 22:52:40.471243 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469256    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 22:52:40.471243 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469259    4294 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 22:52:40.471243 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469262    4294 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 22:52:40.471243 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469264    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 22:52:40.471243 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469267    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 22:52:40.471243 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469269    4294 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 22:52:40.471243 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469272    4294 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 22:52:40.471243 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469275    4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 22:52:40.471799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469277    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 22:52:40.471799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469280    4294 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 22:52:40.471799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469282    4294 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 22:52:40.471799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469285    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 22:52:40.471799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469287    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 22:52:40.471799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469290    4294 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 22:52:40.471799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469293    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 22:52:40.471799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469296    4294 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 22:52:40.471799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469298    4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 22:52:40.471799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469301    4294 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 22:52:40.471799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469303    4294 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 22:52:40.471799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469306    4294 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 22:52:40.471799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469308    4294 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 22:52:40.471799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469311    4294 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 22:52:40.471799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469314    4294 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 22:52:40.471799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469317    4294 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 22:52:40.471799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469319    4294 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 22:52:40.471799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469322    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 22:52:40.471799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469324    4294 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 22:52:40.472313 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469328    4294 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 22:52:40.472313 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469330    4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 22:52:40.472313 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469333    4294 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 22:52:40.472313 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469335    4294 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 22:52:40.472313 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469339    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 22:52:40.472313 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469341    4294 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 22:52:40.472313 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469344    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 22:52:40.472313 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469347    4294 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 22:52:40.472313 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469349    4294 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 22:52:40.472313 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469352    4294 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 22:52:40.472313 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469354    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 22:52:40.472313 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469357    4294 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 22:52:40.472313 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469360    4294 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 22:52:40.472313 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469362    4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 22:52:40.472313 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469366    4294 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 22:52:40.472313 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469370    4294 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 22:52:40.472313 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469373    4294 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 22:52:40.472313 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469376    4294 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 22:52:40.472313 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469379    4294 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 22:52:40.472313 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469381    4294 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 22:52:40.472834 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469385    4294 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 22:52:40.472834 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469387    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 22:52:40.472834 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469390    4294 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 22:52:40.472834 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469393    4294 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 22:52:40.472834 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469396    4294 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 22:52:40.472834 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469398    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 22:52:40.472834 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469401    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 22:52:40.472834 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469403    4294 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 22:52:40.472834 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469406    4294 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 22:52:40.472834 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469409    4294 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 22:52:40.472834 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469411    4294 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 22:52:40.472834 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469414    4294 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 22:52:40.472834 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469417    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 22:52:40.472834 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469420    4294 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 22:52:40.472834 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469423    4294 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 22:52:40.472834 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469425    4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 22:52:40.472834 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469428    4294 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 22:52:40.472834 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469431    4294 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 22:52:40.472834 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469433    4294 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 22:52:40.473337 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469436    4294 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 22:52:40.473337 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469438    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 22:52:40.473337 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469441    4294 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 22:52:40.473337 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469443    4294 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 22:52:40.473337 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469446    4294 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 22:52:40.473337 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469449    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 22:52:40.473337 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469451    4294 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 22:52:40.473337 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469454    4294 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 22:52:40.473337 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469464    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 22:52:40.473337 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469467    4294 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 22:52:40.473337 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469470    4294 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 22:52:40.473337 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469472    4294 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 22:52:40.473337 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469475    4294 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 22:52:40.473337 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469958    4294 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 22:52:40.473337 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469963    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 22:52:40.473337 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469967    4294 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 22:52:40.473337 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469970    4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 22:52:40.473337 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469972    4294 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 22:52:40.473337 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469989    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 22:52:40.473815 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469994    4294 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 22:52:40.473815 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.469997    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 22:52:40.473815 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470005    4294 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 22:52:40.473815 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470008    4294 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 22:52:40.473815 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470011    4294 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 22:52:40.473815 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470014    4294 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 22:52:40.473815 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470017    4294 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 22:52:40.473815 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470020    4294 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 22:52:40.473815 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470023    4294 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 22:52:40.473815 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470025    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 22:52:40.473815 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470028    4294 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 22:52:40.473815 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470031    4294 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 22:52:40.473815 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470034    4294 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 22:52:40.473815 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470036    4294 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 22:52:40.473815 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470039    4294 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 22:52:40.473815 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470042    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 22:52:40.473815 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470044    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 22:52:40.473815 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470047    4294 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 22:52:40.473815 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470050    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 22:52:40.474319 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470052    4294 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 22:52:40.474319 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470055    4294 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 22:52:40.474319 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470057    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 22:52:40.474319 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470060    4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 22:52:40.474319 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470063    4294 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 22:52:40.474319 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470065    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 22:52:40.474319 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470068    4294 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 22:52:40.474319 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470071    4294 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 22:52:40.474319 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470074    4294 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 22:52:40.474319 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470077    4294 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 22:52:40.474319 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470079    4294 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 22:52:40.474319 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470082    4294 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 22:52:40.474319 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470085    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 22:52:40.474319 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470087    4294 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 22:52:40.474319 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470090    4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 22:52:40.474319 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470094    4294 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 22:52:40.474319 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470097    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 22:52:40.474319 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470099    4294 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 22:52:40.474319 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470102    4294 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 22:52:40.474319 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470104    4294 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 22:52:40.474807 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470107    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 22:52:40.474807 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470109    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 22:52:40.474807 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470114    4294 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 22:52:40.474807 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470118    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 22:52:40.474807 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470121    4294 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 22:52:40.474807 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470124    4294 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 22:52:40.474807 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470126    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 22:52:40.474807 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470129    4294 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 22:52:40.474807 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470132    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 22:52:40.474807 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470134    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 22:52:40.474807 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470137    4294 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 22:52:40.474807 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470140    4294 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 22:52:40.474807 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470142    4294 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 22:52:40.474807 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470145    4294 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 22:52:40.474807 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470148    4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 22:52:40.474807 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470150    4294 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 22:52:40.474807 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470153    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 22:52:40.474807 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470155    4294 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 22:52:40.474807 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470158    4294 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 22:52:40.475296 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470160    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 22:52:40.475296 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470163    4294 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 22:52:40.475296 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470166    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 22:52:40.475296 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470169    4294 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 22:52:40.475296 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470171    4294 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 22:52:40.475296 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470174    4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 22:52:40.475296 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470176    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 22:52:40.475296 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470179    4294 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 22:52:40.475296 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470183    4294 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 22:52:40.475296 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470185    4294 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 22:52:40.475296 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470188    4294 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 22:52:40.475296 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470190    4294 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 22:52:40.475296 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470193    4294 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 22:52:40.475296 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470195    4294 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 22:52:40.475296 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470198    4294 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 22:52:40.475296 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470200    4294 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 22:52:40.475296 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470203    4294 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 22:52:40.475296 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470206    4294 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 22:52:40.475296 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470210    4294 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 22:52:40.475768 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470214    4294 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 22:52:40.475768 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470217    4294 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 22:52:40.475768 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470220    4294 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 22:52:40.475768 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470222    4294 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 22:52:40.475768 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470225    4294 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 22:52:40.475768 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470228    4294 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 22:52:40.475768 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470231    4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 22:52:40.475768 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470234    4294 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 22:52:40.475768 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470237    4294 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 22:52:40.475768 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470239    4294 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 22:52:40.475768 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470242    4294 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 22:52:40.475768 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470245    4294 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 22:52:40.475768 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470247    4294 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 22:52:40.475768 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470250    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 22:52:40.475768 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470253    4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 22:52:40.475768 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470255    4294 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 22:52:40.475768 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470258    4294 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 22:52:40.475768 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470261    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 22:52:40.475768 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470264    4294 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 22:52:40.475768 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470267    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470270    4294 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470274    4294 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.470276    4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470370    4294 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470387    4294 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470395    4294 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470399    4294 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470404    4294 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470407    4294 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470411    4294 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470420    4294 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470423    4294 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470426    4294 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470430    4294 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470434    4294 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470437    4294 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470441    4294 flags.go:64] FLAG: --cgroup-root=""
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470445    4294 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470448    4294 flags.go:64] FLAG: --client-ca-file=""
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470451    4294 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470454    4294 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470458    4294 flags.go:64] FLAG: --cluster-domain=""
Nov 02 22:52:40.476291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470461    4294 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470464    4294 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470467    4294 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470470    4294 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470474    4294 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470477    4294 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470480    4294 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470484    4294 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470488    4294 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470491    4294 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470494    4294 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470499    4294 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470502    4294 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470506    4294 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470509    4294 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470512    4294 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470515    4294 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470518    4294 flags.go:64] FLAG: --enable-server="true"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470521    4294 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470526    4294 flags.go:64] FLAG: --event-burst="100"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470529    4294 flags.go:64] FLAG: --event-qps="50"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470532    4294 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470535    4294 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470539    4294 flags.go:64] FLAG: --eviction-hard=""
Nov 02 22:52:40.476838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470543    4294 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470546    4294 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470551    4294 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470554    4294 flags.go:64] FLAG: --eviction-soft=""
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470557    4294 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470560    4294 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470563    4294 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470566    4294 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470569    4294 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470572    4294 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470575    4294 flags.go:64] FLAG: --feature-gates=""
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470579    4294 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470582    4294 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470585    4294 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470589    4294 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470592    4294 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470596    4294 flags.go:64] FLAG: --help="false"
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470599    4294 flags.go:64] FLAG: --hostname-override="ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470603    4294 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470607    4294 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470610    4294 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470614    4294 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470617    4294 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 22:52:40.477493 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470620    4294 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470623    4294 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470626    4294 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470629    4294 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470632    4294 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470635    4294 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470638    4294 flags.go:64] FLAG: --kube-reserved=""
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470641    4294 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470644    4294 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470647    4294 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470649    4294 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470653    4294 flags.go:64] FLAG: --lock-file=""
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470656    4294 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470660    4294 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470662    4294 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470668    4294 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470671    4294 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470674    4294 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470677    4294 flags.go:64] FLAG: --logging-format="text"
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470680    4294 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470683    4294 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470686    4294 flags.go:64] FLAG: --manifest-url=""
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470689    4294 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470693    4294 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470697    4294 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 22:52:40.478076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470700    4294 flags.go:64] FLAG: --max-pods="110"
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470704    4294 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470707    4294 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470711    4294 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470714    4294 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470717    4294 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470720    4294 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470723    4294 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470731    4294 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470734    4294 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470738    4294 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470741    4294 flags.go:64] FLAG: --pod-cidr=""
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470744    4294 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470747    4294 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470750    4294 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470753    4294 flags.go:64] FLAG: --port="10250"
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470756    4294 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470759    4294 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0e400f65c79abf5da"
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470767    4294 flags.go:64] FLAG: --qos-reserved=""
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470770    4294 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470774    4294 flags.go:64] FLAG: --register-node="true"
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470777    4294 flags.go:64] FLAG: --register-with-taints=""
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470781    4294 flags.go:64] FLAG: --registry-burst="10"
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470784    4294 flags.go:64] FLAG: --registry-qps="5"
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470787    4294 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470790    4294 flags.go:64] FLAG: --reserved-memory=""
Nov 02 22:52:40.478702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470793    4294 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470796    4294 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470799    4294 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470803    4294 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470806    4294 flags.go:64] FLAG: --runonce="false"
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470808    4294 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470812    4294 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470815    4294 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470821    4294 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470826    4294 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470829    4294 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470832    4294 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470835    4294 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470838    4294 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470841    4294 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470844    4294 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470847    4294 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470850    4294 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470853    4294 flags.go:64] FLAG: --system-cgroups=""
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470856    4294 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470862    4294 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470864    4294 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470867    4294 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470871    4294 flags.go:64] FLAG: --tls-min-version=""
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470875    4294 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 22:52:40.479374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470878    4294 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470881    4294 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470884    4294 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470887    4294 flags.go:64] FLAG: --v="2"
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470891    4294 flags.go:64] FLAG: --version="false"
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470895    4294 flags.go:64] FLAG: --vmodule=""
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470900    4294 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.470903    4294 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471047    4294 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471053    4294 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471057    4294 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471060    4294 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471063    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471066    4294 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471069    4294 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471072    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471076    4294 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471079    4294 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471082    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471085    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471088    4294 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 22:52:40.479995 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471090    4294 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 22:52:40.480562 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471093    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 22:52:40.480562 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471096    4294 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 22:52:40.480562 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471098    4294 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 22:52:40.480562 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471102    4294 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 22:52:40.480562 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471105    4294 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 22:52:40.480562 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471107    4294 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 22:52:40.480562 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471110    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 22:52:40.480562 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471113    4294 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 22:52:40.480562 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471116    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 22:52:40.480562 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471120    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 22:52:40.480562 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471123    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 22:52:40.480562 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471126    4294 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 22:52:40.480562 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471129    4294 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 22:52:40.480562 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471131    4294 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 22:52:40.480562 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471134    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 22:52:40.480562 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471137    4294 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 22:52:40.480562 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471140    4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 22:52:40.480562 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471142    4294 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 22:52:40.480562 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471145    4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 22:52:40.480562 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471148    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 22:52:40.481390 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471150    4294 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 22:52:40.481390 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471153    4294 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 22:52:40.481390 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471155    4294 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 22:52:40.481390 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471158    4294 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 22:52:40.481390 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471161    4294 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 22:52:40.481390 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471163    4294 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 22:52:40.481390 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471167    4294 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 22:52:40.481390 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471171    4294 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 22:52:40.481390 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471174    4294 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 22:52:40.481390 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471177    4294 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 22:52:40.481390 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471179    4294 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 22:52:40.481390 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471182    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 22:52:40.481390 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471184    4294 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 22:52:40.481390 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471187    4294 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 22:52:40.481390 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471190    4294 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 22:52:40.481390 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471192    4294 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 22:52:40.481390 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471195    4294 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 22:52:40.481390 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471197    4294 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 22:52:40.481390 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471200    4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 22:52:40.481390 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471202    4294 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 22:52:40.482249 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471205    4294 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 22:52:40.482249 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471209    4294 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 22:52:40.482249 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471212    4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 22:52:40.482249 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471214    4294 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 22:52:40.482249 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471217    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 22:52:40.482249 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471219    4294 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 22:52:40.482249 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471222    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 22:52:40.482249 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471225    4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 22:52:40.482249 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471227    4294 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 22:52:40.482249 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471230    4294 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 22:52:40.482249 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471232    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 22:52:40.482249 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471235    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 22:52:40.482249 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471237    4294 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 22:52:40.482249 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471240    4294 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 22:52:40.482249 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471243    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 22:52:40.482249 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471246    4294 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 22:52:40.482249 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471249    4294 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 22:52:40.482249 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471251    4294 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 22:52:40.482249 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471255    4294 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 22:52:40.482831 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471258    4294 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 22:52:40.482831 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471261    4294 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 22:52:40.482831 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471263    4294 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 22:52:40.482831 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471266    4294 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 22:52:40.482831 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471269    4294 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 22:52:40.482831 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471271    4294 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 22:52:40.482831 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471273    4294 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 22:52:40.482831 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471276    4294 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 22:52:40.482831 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471278    4294 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 22:52:40.482831 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471281    4294 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 22:52:40.482831 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471284    4294 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 22:52:40.482831 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471287    4294 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 22:52:40.482831 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471292    4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 22:52:40.482831 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471294    4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 22:52:40.482831 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471299    4294 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 22:52:40.482831 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471302    4294 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 22:52:40.482831 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471305    4294 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 22:52:40.482831 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471314    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 22:52:40.482831 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471317    4294 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 22:52:40.482831 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471320    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 22:52:40.483429 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471322    4294 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 22:52:40.483429 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471325    4294 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 22:52:40.483429 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471327    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 22:52:40.483429 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471330    4294 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 22:52:40.483429 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471333    4294 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 22:52:40.483429 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471335    4294 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 22:52:40.483429 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471338    4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 22:52:40.483429 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471340    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 22:52:40.483429 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471343    4294 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 22:52:40.483429 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471346    4294 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 22:52:40.483429 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471349    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 22:52:40.483429 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471351    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 22:52:40.483429 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471354    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 22:52:40.483429 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.471382    4294 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 22:52:40.483429 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471420    4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 22:52:40.484014 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471425    4294 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 22:52:40.484014 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471429    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 22:52:40.484014 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471432    4294 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 22:52:40.484014 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471435    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 22:52:40.484014 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471437    4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 22:52:40.484014 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471440    4294 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 22:52:40.484014 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471442    4294 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 22:52:40.484014 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471445    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 22:52:40.484014 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471448    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 22:52:40.484014 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471450    4294 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 22:52:40.484014 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471453    4294 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 22:52:40.484014 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471457    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 22:52:40.484014 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471460    4294 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 22:52:40.484014 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471463    4294 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 22:52:40.484014 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471466    4294 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 22:52:40.484014 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471468    4294 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 22:52:40.484014 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471471    4294 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 22:52:40.484014 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471473    4294 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 22:52:40.484014 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471476    4294 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 22:52:40.484690 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471479    4294 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 22:52:40.484690 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471481    4294 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 22:52:40.484690 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471484    4294 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 22:52:40.484690 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471486    4294 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 22:52:40.484690 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471489    4294 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 22:52:40.484690 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471491    4294 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 22:52:40.484690 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471494    4294 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 22:52:40.484690 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471497    4294 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 22:52:40.484690 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471500    4294 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 22:52:40.484690 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471504    4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 22:52:40.484690 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471507    4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 22:52:40.484690 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471509    4294 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 22:52:40.484690 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471512    4294 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 22:52:40.484690 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471515    4294 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 22:52:40.484690 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471518    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 22:52:40.484690 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471520    4294 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 22:52:40.484690 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471523    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 22:52:40.484690 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471525    4294 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 22:52:40.484690 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471528    4294 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 22:52:40.485224 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471530    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 22:52:40.485224 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471533    4294 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 22:52:40.485224 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471535    4294 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 22:52:40.485224 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471538    4294 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 22:52:40.485224 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471540    4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 22:52:40.485224 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471544    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 22:52:40.485224 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471547    4294 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 22:52:40.485224 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471550    4294 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 22:52:40.485224 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471552    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 22:52:40.485224 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471555    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 22:52:40.485224 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471557    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 22:52:40.485224 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471561    4294 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 22:52:40.485224 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471565    4294 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 22:52:40.485224 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471568    4294 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 22:52:40.485224 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471571    4294 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 22:52:40.485224 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471574    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 22:52:40.485224 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471576    4294 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 22:52:40.485224 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471579    4294 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 22:52:40.485224 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471582    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 22:52:40.485732 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471585    4294 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 22:52:40.485732 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471587    4294 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 22:52:40.485732 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471590    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 22:52:40.485732 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471592    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 22:52:40.485732 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471595    4294 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 22:52:40.485732 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471598    4294 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 22:52:40.485732 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471600    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 22:52:40.485732 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471604    4294 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 22:52:40.485732 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471607    4294 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 22:52:40.485732 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471610    4294 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 22:52:40.485732 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471612    4294 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 22:52:40.485732 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471615    4294 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 22:52:40.485732 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471617    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 22:52:40.485732 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471620    4294 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 22:52:40.485732 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471622    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 22:52:40.485732 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471625    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 22:52:40.485732 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471628    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 22:52:40.485732 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471630    4294 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 22:52:40.485732 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471634    4294 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 22:52:40.485732 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471637    4294 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 22:52:40.486409 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471639    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 22:52:40.486409 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471642    4294 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 22:52:40.486409 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471644    4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 22:52:40.486409 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471647    4294 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 22:52:40.486409 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471649    4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 22:52:40.486409 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471652    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 22:52:40.486409 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471654    4294 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 22:52:40.486409 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471657    4294 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 22:52:40.486409 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471659    4294 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 22:52:40.486409 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471664    4294 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 22:52:40.486409 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471667    4294 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 22:52:40.486409 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471670    4294 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 22:52:40.486409 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471673    4294 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 22:52:40.486409 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471675    4294 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 22:52:40.486409 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471678    4294 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 22:52:40.486409 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471680    4294 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 22:52:40.486409 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471683    4294 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 22:52:40.486409 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471686    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 22:52:40.486409 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471688    4294 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 22:52:40.486409 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471706    4294 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 22:52:40.486897 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471711    4294 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 22:52:40.486897 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471714    4294 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 22:52:40.486897 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471718    4294 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 22:52:40.486897 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471721    4294 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 22:52:40.486897 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471724    4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 22:52:40.486897 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471727    4294 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 22:52:40.486897 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471730    4294 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 22:52:40.486897 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.471732    4294 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 22:52:40.486897 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.475465    4294 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 22:52:40.486897 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.475482    4294 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 22:52:40.486897 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.475500    4294 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 22:52:40.486897 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.475505    4294 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 22:52:40.486897 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475563    4294 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 22:52:40.486897 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475569    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 22:52:40.486897 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475572    4294 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 22:52:40.486897 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475576    4294 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 22:52:40.486897 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475579    4294 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 22:52:40.486897 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475582    4294 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 22:52:40.486897 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475585    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 22:52:40.486897 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475587    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 22:52:40.487456 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475590    4294 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 22:52:40.487456 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475593    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 22:52:40.487456 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475596    4294 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 22:52:40.487456 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475599    4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 22:52:40.487456 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475602    4294 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 22:52:40.487456 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475605    4294 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 22:52:40.487456 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475608    4294 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 22:52:40.487456 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475611    4294 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 22:52:40.487456 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475613    4294 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 22:52:40.487456 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475616    4294 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 22:52:40.487456 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475619    4294 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 22:52:40.487456 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475621    4294 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 22:52:40.487456 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475624    4294 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 22:52:40.487456 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475627    4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 22:52:40.487456 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475630    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 22:52:40.487456 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475632    4294 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 22:52:40.487456 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475636    4294 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 22:52:40.487456 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475639    4294 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 22:52:40.487456 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475642    4294 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 22:52:40.487926 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475644    4294 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 22:52:40.487926 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475647    4294 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 22:52:40.487926 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475651    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 22:52:40.487926 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475654    4294 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 22:52:40.487926 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475656    4294 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 22:52:40.487926 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475659    4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 22:52:40.487926 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475662    4294 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 22:52:40.487926 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475664    4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 22:52:40.487926 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475667    4294 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 22:52:40.487926 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475669    4294 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 22:52:40.487926 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475672    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 22:52:40.487926 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475675    4294 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 22:52:40.487926 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475677    4294 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 22:52:40.487926 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475680    4294 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 22:52:40.487926 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475683    4294 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 22:52:40.487926 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475686    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 22:52:40.487926 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475688    4294 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 22:52:40.487926 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475691    4294 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 22:52:40.487926 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475694    4294 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 22:52:40.488418 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475697    4294 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 22:52:40.488418 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475699    4294 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 22:52:40.488418 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475702    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 22:52:40.488418 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475705    4294 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 22:52:40.488418 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475707    4294 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 22:52:40.488418 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475710    4294 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 22:52:40.488418 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475713    4294 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 22:52:40.488418 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475716    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 22:52:40.488418 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475719    4294 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 22:52:40.488418 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475721    4294 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 22:52:40.488418 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475724    4294 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 22:52:40.488418 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475727    4294 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 22:52:40.488418 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475730    4294 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 22:52:40.488418 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475734    4294 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 22:52:40.488418 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475736    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 22:52:40.488418 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475738    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 22:52:40.488418 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475742    4294 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 22:52:40.488418 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475746    4294 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 22:52:40.488418 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475749    4294 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 22:52:40.488418 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475753    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 22:52:40.488888 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475756    4294 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 22:52:40.488888 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475759    4294 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 22:52:40.488888 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475761    4294 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 22:52:40.488888 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475764    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 22:52:40.488888 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475767    4294 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 22:52:40.488888 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475769    4294 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 22:52:40.488888 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475772    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 22:52:40.488888 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475776    4294 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 22:52:40.488888 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475778    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 22:52:40.488888 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475781    4294 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 22:52:40.488888 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475784    4294 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 22:52:40.488888 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475787    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 22:52:40.488888 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475789    4294 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 22:52:40.488888 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475792    4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 22:52:40.488888 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475794    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 22:52:40.488888 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475797    4294 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 22:52:40.488888 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475800    4294 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 22:52:40.488888 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475803    4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 22:52:40.488888 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475805    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 22:52:40.488888 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475808    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 22:52:40.489386 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475811    4294 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 22:52:40.489386 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475814    4294 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 22:52:40.489386 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475817    4294 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 22:52:40.489386 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475819    4294 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 22:52:40.489386 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475822    4294 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 22:52:40.489386 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475825    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 22:52:40.489386 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475828    4294 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 22:52:40.489386 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475845    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 22:52:40.489386 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475849    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 22:52:40.489386 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475853    4294 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 22:52:40.489386 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475856    4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 22:52:40.489386 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475859    4294 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 22:52:40.489386 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475862    4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 22:52:40.489386 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475864    4294 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 22:52:40.489386 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475867    4294 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 22:52:40.489386 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475870    4294 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 22:52:40.489386 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475872    4294 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 22:52:40.489386 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475875    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 22:52:40.489386 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475879    4294 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 22:52:40.489850 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.475883    4294 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 22:52:40.489850 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.475909    4294 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 22:52:40.489850 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476072    4294 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 22:52:40.489850 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476077    4294 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 22:52:40.489850 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476080    4294 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 22:52:40.489850 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476083    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 22:52:40.489850 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476086    4294 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 22:52:40.489850 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476089    4294 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 22:52:40.489850 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476093    4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 22:52:40.489850 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476095    4294 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 22:52:40.489850 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476098    4294 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 22:52:40.489850 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476101    4294 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 22:52:40.489850 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476104    4294 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 22:52:40.489850 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476108    4294 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 22:52:40.489850 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476111    4294 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 22:52:40.490250 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476114    4294 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 22:52:40.490250 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476118    4294 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 22:52:40.490250 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476121    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 22:52:40.490250 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476123    4294 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 22:52:40.490250 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476126    4294 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 22:52:40.490250 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476129    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 22:52:40.490250 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476131    4294 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 22:52:40.490250 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476134    4294 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 22:52:40.490250 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476138    4294 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 22:52:40.490250 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476140    4294 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 22:52:40.490250 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476143    4294 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 22:52:40.490250 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476146    4294 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 22:52:40.490250 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476148    4294 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 22:52:40.490250 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476151    4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 22:52:40.490250 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476153    4294 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 22:52:40.490250 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476156    4294 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 22:52:40.490250 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476158    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 22:52:40.490250 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476161    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 22:52:40.490250 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476163    4294 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 22:52:40.490250 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476166    4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 22:52:40.490723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476168    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 22:52:40.490723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476171    4294 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 22:52:40.490723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476173    4294 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 22:52:40.490723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476176    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 22:52:40.490723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476178    4294 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 22:52:40.490723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476181    4294 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 22:52:40.490723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476184    4294 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 22:52:40.490723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476187    4294 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 22:52:40.490723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476189    4294 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 22:52:40.490723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476192    4294 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 22:52:40.490723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476194    4294 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 22:52:40.490723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476197    4294 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 22:52:40.490723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476199    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 22:52:40.490723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476202    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 22:52:40.490723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476205    4294 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 22:52:40.490723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476208    4294 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 22:52:40.490723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476210    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 22:52:40.490723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476213    4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 22:52:40.490723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476216    4294 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 22:52:40.490723 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476218    4294 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 22:52:40.491218 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476221    4294 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 22:52:40.491218 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476224    4294 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 22:52:40.491218 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476227    4294 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 22:52:40.491218 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476229    4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 22:52:40.491218 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476232    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 22:52:40.491218 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476235    4294 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 22:52:40.491218 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476237    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 22:52:40.491218 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476240    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 22:52:40.491218 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476242    4294 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 22:52:40.491218 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476245    4294 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 22:52:40.491218 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476248    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 22:52:40.491218 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476250    4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 22:52:40.491218 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476253    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 22:52:40.491218 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476255    4294 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 22:52:40.491218 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476258    4294 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 22:52:40.491218 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476261    4294 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 22:52:40.491218 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476263    4294 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 22:52:40.491218 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476266    4294 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 22:52:40.491218 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476269    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 22:52:40.491685 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476273    4294 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 22:52:40.491685 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476276    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 22:52:40.491685 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476280    4294 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 22:52:40.491685 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476284    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 22:52:40.491685 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476286    4294 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 22:52:40.491685 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476289    4294 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 22:52:40.491685 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476292    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 22:52:40.491685 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476295    4294 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 22:52:40.491685 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476298    4294 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 22:52:40.491685 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476301    4294 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 22:52:40.491685 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476303    4294 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 22:52:40.491685 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476306    4294 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 22:52:40.491685 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476309    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 22:52:40.491685 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476311    4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 22:52:40.491685 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476314    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 22:52:40.491685 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476317    4294 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 22:52:40.491685 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476320    4294 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 22:52:40.491685 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476322    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 22:52:40.491685 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476325    4294 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 22:52:40.492160 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476327    4294 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 22:52:40.492160 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476330    4294 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 22:52:40.492160 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476333    4294 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 22:52:40.492160 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476335    4294 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 22:52:40.492160 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476338    4294 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 22:52:40.492160 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476340    4294 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 22:52:40.492160 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476342    4294 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 22:52:40.492160 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476345    4294 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 22:52:40.492160 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476348    4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 22:52:40.492160 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476351    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 22:52:40.492160 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476353    4294 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 22:52:40.492160 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476356    4294 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 22:52:40.492160 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476359    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 22:52:40.492160 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476375    4294 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 22:52:40.492160 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476380    4294 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 22:52:40.492160 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.476403    4294 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 22:52:40.492677 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476437    4294 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 22:52:40.492677 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476441    4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 22:52:40.492677 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476443    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 22:52:40.492677 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476446    4294 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 22:52:40.492677 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476449    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 22:52:40.492677 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476452    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 22:52:40.492677 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476455    4294 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 22:52:40.492677 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476457    4294 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 22:52:40.492677 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476460    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 22:52:40.492677 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476462    4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 22:52:40.492677 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476465    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 22:52:40.492677 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476467    4294 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 22:52:40.492677 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476470    4294 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 22:52:40.492677 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476473    4294 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 22:52:40.492677 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476476    4294 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 22:52:40.492677 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476479    4294 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 22:52:40.492677 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476481    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 22:52:40.492677 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476485    4294 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 22:52:40.492677 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476489    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 22:52:40.493397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476492    4294 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 22:52:40.493397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476494    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 22:52:40.493397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476497    4294 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 22:52:40.493397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476500    4294 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 22:52:40.493397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476502    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 22:52:40.493397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476505    4294 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 22:52:40.493397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476507    4294 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 22:52:40.493397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476510    4294 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 22:52:40.493397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476513    4294 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 22:52:40.493397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476515    4294 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 22:52:40.493397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476518    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 22:52:40.493397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476521    4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 22:52:40.493397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476523    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 22:52:40.493397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476526    4294 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 22:52:40.493397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476529    4294 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 22:52:40.493397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476531    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 22:52:40.493397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476538    4294 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 22:52:40.493397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476541    4294 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 22:52:40.493397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476544    4294 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 22:52:40.493397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476546    4294 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 22:52:40.494005 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476549    4294 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 22:52:40.494005 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476552    4294 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 22:52:40.494005 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476554    4294 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 22:52:40.494005 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476557    4294 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 22:52:40.494005 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476560    4294 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 22:52:40.494005 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476563    4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 22:52:40.494005 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476565    4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 22:52:40.494005 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476568    4294 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 22:52:40.494005 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476571    4294 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 22:52:40.494005 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476573    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 22:52:40.494005 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476576    4294 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 22:52:40.494005 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476579    4294 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 22:52:40.494005 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476581    4294 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 22:52:40.494005 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476584    4294 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 22:52:40.494005 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476587    4294 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 22:52:40.494005 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476591    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 22:52:40.494005 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476593    4294 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 22:52:40.494005 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476596    4294 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 22:52:40.494005 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476599    4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 22:52:40.494799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476601    4294 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 22:52:40.494799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476604    4294 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 22:52:40.494799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476606    4294 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 22:52:40.494799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476609    4294 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 22:52:40.494799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476613    4294 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 22:52:40.494799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476618    4294 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 22:52:40.494799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476621    4294 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 22:52:40.494799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476623    4294 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 22:52:40.494799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476626    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 22:52:40.494799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476630    4294 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 22:52:40.494799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476632    4294 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 22:52:40.494799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476635    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 22:52:40.494799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476638    4294 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 22:52:40.494799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476640    4294 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 22:52:40.494799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476643    4294 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 22:52:40.494799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476645    4294 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 22:52:40.494799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476648    4294 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 22:52:40.494799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476650    4294 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 22:52:40.494799 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476653    4294 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 22:52:40.495847 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476656    4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 22:52:40.495847 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476658    4294 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 22:52:40.495847 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476662    4294 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 22:52:40.495847 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476664    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 22:52:40.495847 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476667    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 22:52:40.495847 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476670    4294 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 22:52:40.495847 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476672    4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 22:52:40.495847 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476675    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 22:52:40.495847 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476677    4294 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 22:52:40.495847 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476680    4294 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 22:52:40.495847 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476683    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 22:52:40.495847 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476685    4294 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 22:52:40.495847 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476688    4294 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 22:52:40.495847 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476691    4294 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 22:52:40.495847 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476693    4294 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 22:52:40.495847 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476696    4294 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 22:52:40.495847 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476698    4294 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 22:52:40.495847 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476701    4294 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 22:52:40.495847 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476704    4294 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 22:52:40.495847 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476706    4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 22:52:40.496745 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476709    4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 22:52:40.496745 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476711    4294 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 22:52:40.496745 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476716    4294 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 22:52:40.496745 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476718    4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 22:52:40.496745 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476722    4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 22:52:40.496745 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476725    4294 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 22:52:40.496745 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476728    4294 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 22:52:40.496745 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476730    4294 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 22:52:40.496745 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:40.476733    4294 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 22:52:40.496745 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.476782    4294 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 22:52:40.496745 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.477769    4294 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 22:52:40.496745 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.477818    4294 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 22:52:40.496745 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.478016    4294 server.go:1014] "Starting client certificate rotation"
Nov 02 22:52:40.496745 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.478163    4294 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 22:52:40.496745 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.478219    4294 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 22:47:01 +0000 UTC" deadline="2028-06-21 22:10:22.465426855 +0000 UTC"
Nov 02 22:52:40.496745 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.478264    4294 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14327h17m41.987165499s"
Nov 02 22:52:40.497453 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.478536    4294 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 22:52:40.497453 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.479691    4294 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 22:52:40.497453 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.482148    4294 log.go:25] "Validated CRI v1 runtime API"
Nov 02 22:52:40.497453 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.483703    4294 log.go:25] "Validated CRI v1 image API"
Nov 02 22:52:40.497453 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.484425    4294 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 22:52:40.497453 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.485773    4294 fs.go:135] Filesystem UUIDs: map[3c7acda2-d5e0-49c6-afa4-2d41cb8a711a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e93a8b64-fb10-4893-a847-7c0dcf7ceca1:/dev/nvme0n1p3]
Nov 02 22:52:40.497734 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.485795    4294 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2dcd50348bcf9c60f8141439ad5bc51f5e57c17db613473548270e0ebaa46be2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2dcd50348bcf9c60f8141439ad5bc51f5e57c17db613473548270e0ebaa46be2/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87fe95f3eb83273324f4440483aac39cb7cf35a79c053349b628fb609ed9c00d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87fe95f3eb83273324f4440483aac39cb7cf35a79c053349b628fb609ed9c00d/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8bc543e55ddf52e0658042f60d9155c4ccb6bf8c7497816d696f46aab1f64bea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8bc543e55ddf52e0658042f60d9155c4ccb6bf8c7497816d696f46aab1f64bea/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91450f8b91b9d1ab9ef1e553b441595f4093d010afd7144301deb6db46924170/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91450f8b91b9d1ab9ef1e553b441595f4093d010afd7144301deb6db46924170/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/943ab21fc7bc5cc5287fafd80dd31f5df3efa877190b6a6cbdb8d94ba508d2b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/943ab21fc7bc5cc5287fafd80dd31f5df3efa877190b6a6cbdb8d94ba508d2b3/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95d2e09830c2b2560a1284968047f359c1c9a54041e06a7db62d38e4d73ae8e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95d2e09830c2b2560a1284968047f359c1c9a54041e06a7db62d38e4d73ae8e7/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d11b1994899cb37357fb0763c08a84a782e32827d09bd6049b89c1023e3ba60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d11b1994899cb37357fb0763c08a84a782e32827d09bd6049b89c1023e3ba60/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a15ab56b9280ee70cea0943cf2842d69055df959b7060863e107f4fd75e0c7b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a15ab56b9280ee70cea0943cf2842d69055df959b7060863e107f4fd75e0c7b2/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7e7a4e327a441143ba1a60932f1b143231e7f12758ffaa6134e82b54ad4b467/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7e7a4e327a441143ba1a60932f1b143231e7f12758ffaa6134e82b54ad4b467/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae50de4dcc40e8a529963141abb98c14f84a45dc0414b2cf83fc93745d2d6284/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae50de4dcc40e8a529963141abb98c14f84a45dc0414b2cf83fc93745d2d6284/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5580e7c20fe728be4563a67ca5e2e8a3164967866d28f8231960313f313aa9e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5580e7c20fe728be4563a67ca5e2e8a3164967866d28f8231960313f313aa9e/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cbd4a1fd00f652af41d91fc4b9d669d983a3974c6c5f1509a11389c197b04e23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cbd4a1fd00f652af41d91fc4b9d669d983a3974c6c5f1509a11389c197b04e23/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ce80856-f167-427e-b8d5-4b856bfcf1ea/volumes/kubernetes.io~projected/kube-api-access-r6hww:{mountpoint:/var/lib/kubelet/pods/0ce80856-f167-427e-b8d5-4b856bfcf1ea/volumes/kubernetes.io~projected/kube-api-access-r6hww major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11efd367-1b70-4cb5-93c5-2dda7b9bb010/volumes/kubernetes.io~projected/kube-api-access-nxxqd:{mountpoint:/var/lib/kubelet/pods/11efd367-1b70-4cb5-93c5-2dda7b9bb010/volumes/kubernetes.io~projected/kube-api-access-nxxqd major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18dcacc0-751d-4f2e-8101-59a1da48fab2/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/18dcacc0-751d-4f2e-8101-59a1da48fab2/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18dcacc0-751d-4f2e-8101-59a1da48fab2/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/18dcacc0-751d-4f2e-8101-59a1da48fab2/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18dcacc0-751d-4f2e-8101-59a1da48fab2/volumes/kubernetes.io~projected/kube-api-access-g9zml:{mountpoint:/var/lib/kubelet/pods/18dcacc0-751d-4f2e-8101-59a1da48fab2/volumes/kubernetes.io~projected/kube-api-access-g9zml major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a474267-46b1-4759-9018-54960f751878/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/3a474267-46b1-4759-9018-54960f751878/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b1cf92b-28eb-4894-96ed-b7eba4a781d2/volumes/kubernetes.io~projected/kube-api-access-25fvj:{mountpoint:/var/lib/kubelet/pods/4b1cf92b-28eb-4894-96ed-b7eba4a781d2/volumes/kubernetes.io~projected/kube-api-access-25fvj major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b8c537b-69a3-495b-a99b-afcbdc20ddf6/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/5b8c537b-69a3-495b-a99b-afcbdc20ddf6/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b8c537b-69a3-495b-a99b-afcbdc20ddf6/volumes/kubernetes.io~projected/kube-api-access-456qc:{mountpoint:/var/lib/kubelet/pods/5b8c537b-69a3-495b-a99b-afcbdc20ddf6/volumes/kubernetes.io~projected/kube-api-access-456qc major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b8c537b-69a3-495b-a99b-afcbdc20ddf6/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/5b8c537b-69a3-495b-a99b-afcbdc20ddf6/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8161cb48-56bb-455b-bee6-cd3d06607109/volumes/kubernetes.io~projected/kube-api-access-drxcv:{mountpoint:/var/lib/kubelet/pods/8161cb48-56bb-455b-bee6-cd3d06607109/volumes/kubernetes.io~projected/kube-api-access-drxcv major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e/volumes/kubernetes.io~projected/kube-api-access-x9m47:{mountpoint:/var/lib/kubelet/pods/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e/volumes/kubernetes.io~projected/kube-api-access-x9m47 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92845990-417f-4c8f-aa67-ea9961516288/volumes/kubernetes.io~projected/kube-api-access-c4nhs:{mountpoint:/var/lib/kubelet/pods/92845990-417f-4c8f-aa67-ea9961516288/volumes/kubernetes.io~projected/kube-api-access-c4nhs major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1bd54f8-715f-428f-998b-65c6e7f8830e/volumes/kubernetes.io~projected/kube-api-access-5hl8b:{mountpoint:/var/lib/kubelet/pods/b1bd54f8-715f-428f-998b-65c6e7f8830e/volumes/kubernetes.io~projected/kube-api-access-5hl8b major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e37c6a5d-1170-4f2d-ae18-2580da36f53b/volumes/kubernetes.io~projected/kube-api-access-j9qmn:{mountpoint:/var/lib/kubelet/pods/e37c6a5d-1170-4f2d-ae18-2580da36f53b/volumes/kubernetes.io~projected/kube-api-access-j9qmn major:0 minor:54 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/2a9666dc179bc9e84a1a3b89413963e646ef1d09722154d3f7bf1dd0761dbcef/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/dd123ce8c8578ca782473478d34ffef49adaf34dcdc0421308031c856492b23b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/8b1f1a553733b983ec7be7ef9f5e7d7ef2b22e3fcc6f21dd639321fe154bb9f2/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/cc9e08b9a35829c0512fdf58d4c653180e0b01d014a478c40175e1f5fe2bbc8e/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b192445d952909406f25eb25ff5184b1f524e519e9d292be6b0c8d8302169a1d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/156595057b15848c3603a3585d1f7a45bb49be660758f6a192d9da1dc0ea2c6c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-121:{mountpoint:/var/lib/containers/storage/overlay/8fd16c94eff64a7149b43a880d52e1ca027972f5da2f3cc30718cc919feac6c9/merged major:0 minor:121 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8e77e1d7c8cff118d6927336d74a6fd77f0e63974662943499ebe932710d5477/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/2dd5b5798608ca7328c3f8c4fbeb780448599ef6e3eb7c0dc918d165e3f8af3c/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/0eca765522c592f9e871ea2426155037215aaa16067c6fd434f91d4261764c0f/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/545548b8eb02b2b90892906c0a329b91d202400c0571683f0ca4a07f2c190ae5/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/3942701eb5e42b38f85297ad732c9808e2db11f500f7e96d21e87e0e59256a0c/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/59af49aac9a75c729221cbc817f2164b03943d22703c87b202a18c82631f43af/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/c44bfcd359ad7d165794cf12d5dc7e68659d60daff0ba17354ceefa5755e922c/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/1ca686174ff515dae32460edc44925df80d19ee6f332c736f0a5368a2f806c03/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/6702853ea2b8d4e0aaaeb1bf9413ed744cadebec98a42f8c1ca574fd5d30a2c4/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/b30c0011fa92adbbccf19e4873a998c9e3b9ff58a4ae6d277a6442d6552fd1ba/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/382be8e8a94d6b348275e8d66c862e3890f91738235727be132f9f69616dca90/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/03798acbb0dea4fe87f7c469d443e91496d2a03a7fd28b2a7d93f687d62a548e/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/bae06a7bacb228b1a724dfa916eae28e7c2591923dac8f2086d2ec5c2cada76f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/194fad0c414f053008715fda90b4675d5d67fc4d4d3efa1e3ce0216a1545bad0/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/0b821642eec781cc0f60ba9cca92ef21daee367fd13fcab2195dd860f7052326/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/ddef9e24765688a84a15a14bb4e83d0d87dcfc08a46b835f09aba00e1bf50685/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/d7bc89183cc32d4b42c1ddf0133ba97fa360410d71b5242c22eddf4537d8bb52/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/eb6a4596e45490cf49156af87923520e46df71ed3bf36deb2479636208335a93/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/6766e7c20fe22e8afcfd931be85852a3f412cbb23f514e4e2638a3a5748171ca/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/dd7cb0212b128f85fa3d49f50468ccc3023ddcb334cc918576d5e6596bc36c2d/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/2ecfe880757f6e3734a394c504f65a0d55544e84e4e8630989927e8a5b3818d7/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/8db55fdd03810ad930d0d2a470dda92bb4adf9d49f9006fe0b2663d4e63952f8/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/e31360e5093b72b34ef6eac316c8253eacc80c380f90b151fd3a4b8df19cffe6/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/a75065c5bcec6085d5172783ae02331d001abe29948e3afb93cf6d2f4c4e8e2b/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/64e6ef490ecc92c59661c59854b8d5c384cf3854c069a3da8f3b7eceabf4250f/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/0db55e91a69aa05a24f8b1bb39fd1b0ca65fee44d083b64988f4f5374ae12cc2/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/e1d6800f59ab6e5504d7f1eed75c8c2c099feaff4c86914208feab2cb7757828/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/8f6bf3139a6765ecf20f0158db462c3fca2c7eca3b8fd59b375d4dfc7a494292/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 22:52:40.497734 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.491255    4294 manager.go:217] Machine: {Timestamp:2026-11-02 22:52:40.490785647 +0000 UTC m=+0.059140424 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100364 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dde3fe5e593041dc59bf425971cf8 SystemUUID:ec2dde3f-e5e5-9304-1dc5-9bf425971cf8 BootID:1b9479a7-4fc0-4a45-a5b4-65f211d57cdd Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/18dcacc0-751d-4f2e-8101-59a1da48fab2/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0ce80856-f167-427e-b8d5-4b856bfcf1ea/volumes/kubernetes.io~projected/kube-api-access-r6hww DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87fe95f3eb83273324f4440483aac39cb7cf35a79c053349b628fb609ed9c00d/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d11b1994899cb37357fb0763c08a84a782e32827d09bd6049b89c1023e3ba60/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3a474267-46b1-4759-9018-54960f751878/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e/volumes/kubernetes.io~projected/kube-api-access-x9m47 DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92845990-417f-4c8f-aa67-ea9961516288/volumes/kubernetes.io~projected/kube-api-access-c4nhs DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5b8c537b-69a3-495b-a99b-afcbdc20ddf6/volumes/kubernetes.io~projected/kube-api-access-456qc DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8bc543e55ddf52e0658042f60d9155c4ccb6bf8c7497816d696f46aab1f64bea/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e37c6a5d-1170-4f2d-ae18-2580da36f53b/volumes/kubernetes.io~projected/kube-api-access-j9qmn DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1bd54f8-715f-428f-998b-65c6e7f8830e/volumes/kubernetes.io~projected/kube-api-access-5hl8b DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7e7a4e327a441143ba1a60932f1b143231e7f12758ffaa6134e82b54ad4b467/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91450f8b91b9d1ab9ef1e553b441595f4093d010afd7144301deb6db46924170/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2dcd50348bcf9c60f8141439ad5bc51f5e57c17db613473548270e0ebaa46be2/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/943ab21fc7bc5cc5287fafd80dd31f5df3efa877190b6a6cbdb8d94ba508d2b3/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8161cb48-56bb-455b-bee6-cd3d06607109/volumes/kubernetes.io~projected/kube-api-access-drxcv DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-121 DeviceMajor:0 DeviceMinor:121 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11efd367-1b70-4cb5-93c5-2dda7b9bb010/volumes/kubernetes.io~projected/kube-api-access-nxxqd DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5580e7c20fe728be4563a67ca5e2e8a3164967866d28f8231960313f313aa9e/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b8c537b-69a3-495b-a99b-afcbdc20ddf6/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cbd4a1fd00f652af41d91fc4b9d669d983a3974c6c5f1509a11389c197b04e23/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18dcacc0-751d-4f2e-8101-59a1da48fab2/volumes/kubernetes.io~projected/kube-api-access-g9zml DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95d2e09830c2b2560a1284968047f359c1c9a54041e06a7db62d38e4d73ae8e7/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4b1cf92b-28eb-4894-96ed-b7eba4a781d2/volumes/kubernetes.io~projected/kube-api-access-25fvj DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b8c537b-69a3-495b-a99b-afcbdc20ddf6/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18dcacc0-751d-4f2e-8101-59a1da48fab2/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a15ab56b9280ee70cea0943cf2842d69055df959b7060863e107f4fd75e0c7b2/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae50de4dcc40e8a529963141abb98c14f84a45dc0414b2cf83fc93745d2d6284/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:8bc543e55ddf52e MacAddress:d2:c3:60:50:64:2a Speed:10000 Mtu:8901} {Name:95d2e09830c2b25 MacAddress:12:80:55:45:85:88 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:e4:77:29:ab Speed:0 Mtu:9001} {Name:br-int MacAddress:62:b5:4f:f5:d2:46 Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:e4:77:29:ab Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:b2:db:b1:4d:57:11 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:16:6d:82:40:71:90 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 22:52:40.497734 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.491488    4294 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 22:52:40.497734 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.491531    4294 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 22:52:40.497734 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.491806    4294 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 22:52:40.497734 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.491830    4294 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-234.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 22:52:40.497734 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.491952    4294 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 22:52:40.497734 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.491960    4294 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 22:52:40.497734 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.491974    4294 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 22:52:40.497734 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.492005    4294 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 22:52:40.497734 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.492088    4294 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 22:52:40.497734 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.492239    4294 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 22:52:40.497734 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.492338    4294 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 22:52:40.497734 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.492371    4294 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 22:52:40.497734 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.492384    4294 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.492394    4294 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.492405    4294 kubelet.go:405] "Adding apiserver pod source"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.492413    4294 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.493492    4294 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.493502    4294 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.493960    4294 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.496515    4294 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.496528    4294 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.496636    4294 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.496899    4294 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.496944    4294 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.497076    4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.497094    4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.497103    4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.497113    4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.497122    4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.497132    4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.497143    4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.497152    4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.497162    4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.497172    4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.497186    4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.497199    4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.497228    4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 22:52:40.498727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.497239    4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 22:52:40.501187 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.501170    4294 server.go:1263] "Started kubelet"
Nov 02 22:52:40.501288 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.501218    4294 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 22:52:40.501655 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.501379    4294 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 22:52:40.501655 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.501455    4294 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 22:52:40.501703 ip-10-0-139-234 systemd[1]: Started Kubernetes Kubelet.
Nov 02 22:52:40.502841 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.502292    4294 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 22:52:40.504253 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.504092    4294 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 22:52:40.506118 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.506103    4294 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 22:52:40.506242 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.506196    4294 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 22:52:40.507357 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.506320    4294 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 22:47:02 +0000 UTC" deadline="2028-08-04 09:17:07.91711284 +0000 UTC"
Nov 02 22:52:40.507357 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.506337    4294 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 22:52:40.507357 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.506340    4294 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15370h24m27.410775782s"
Nov 02 22:52:40.507357 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.506347    4294 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 22:52:40.507357 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:52:40.506389    4294 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-234.us-west-2.compute.internal\" not found"
Nov 02 22:52:40.507357 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.506407    4294 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507492    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e71e09c-79f2-42d0-a2ee-e1535cf3df8e" volumeName="kubernetes.io/projected/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e-kube-api-access-x9m47" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507525    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92845990-417f-4c8f-aa67-ea9961516288" volumeName="kubernetes.io/empty-dir/92845990-417f-4c8f-aa67-ea9961516288-tmp-dir" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507541    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18dcacc0-751d-4f2e-8101-59a1da48fab2" volumeName="kubernetes.io/projected/18dcacc0-751d-4f2e-8101-59a1da48fab2-kube-api-access-g9zml" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507554    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92845990-417f-4c8f-aa67-ea9961516288" volumeName="kubernetes.io/projected/92845990-417f-4c8f-aa67-ea9961516288-kube-api-access-c4nhs" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507566    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1bd54f8-715f-428f-998b-65c6e7f8830e" volumeName="kubernetes.io/configmap/b1bd54f8-715f-428f-998b-65c6e7f8830e-cni-binary-copy" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507578    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e37c6a5d-1170-4f2d-ae18-2580da36f53b" volumeName="kubernetes.io/projected/e37c6a5d-1170-4f2d-ae18-2580da36f53b-kube-api-access-j9qmn" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507591    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ce80856-f167-427e-b8d5-4b856bfcf1ea" volumeName="kubernetes.io/configmap/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-daemon-config" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507605    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0dfa862-aaf8-4ff1-820e-a610a4d41ba8" volumeName="kubernetes.io/configmap/a0dfa862-aaf8-4ff1-820e-a610a4d41ba8-konnectivity-ca" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507621    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0dfa862-aaf8-4ff1-820e-a610a4d41ba8" volumeName="kubernetes.io/secret/a0dfa862-aaf8-4ff1-820e-a610a4d41ba8-agent-certs" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507633    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1bd54f8-715f-428f-998b-65c6e7f8830e" volumeName="kubernetes.io/configmap/b1bd54f8-715f-428f-998b-65c6e7f8830e-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507645    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1bd54f8-715f-428f-998b-65c6e7f8830e" volumeName="kubernetes.io/configmap/b1bd54f8-715f-428f-998b-65c6e7f8830e-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507658    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1bd54f8-715f-428f-998b-65c6e7f8830e" volumeName="kubernetes.io/projected/b1bd54f8-715f-428f-998b-65c6e7f8830e-kube-api-access-5hl8b" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507687    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e37c6a5d-1170-4f2d-ae18-2580da36f53b" volumeName="kubernetes.io/configmap/e37c6a5d-1170-4f2d-ae18-2580da36f53b-iptables-alerter-script" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507699    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ce80856-f167-427e-b8d5-4b856bfcf1ea" volumeName="kubernetes.io/projected/0ce80856-f167-427e-b8d5-4b856bfcf1ea-kube-api-access-r6hww" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507730    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18dcacc0-751d-4f2e-8101-59a1da48fab2" volumeName="kubernetes.io/empty-dir/18dcacc0-751d-4f2e-8101-59a1da48fab2-tmp" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507744    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18dcacc0-751d-4f2e-8101-59a1da48fab2" volumeName="kubernetes.io/empty-dir/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-tuned" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507757    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a474267-46b1-4759-9018-54960f751878" volumeName="kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507768    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b1cf92b-28eb-4894-96ed-b7eba4a781d2" volumeName="kubernetes.io/projected/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-kube-api-access-25fvj" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507781    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b8c537b-69a3-495b-a99b-afcbdc20ddf6" volumeName="kubernetes.io/configmap/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-ovnkube-config" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507786    4294 factory.go:55] Registering systemd factory
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507793    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b8c537b-69a3-495b-a99b-afcbdc20ddf6" volumeName="kubernetes.io/secret/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507808    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8161cb48-56bb-455b-bee6-cd3d06607109" volumeName="kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507819    4294 factory.go:223] Registration of the systemd container factory successfully
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507821    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ce80856-f167-427e-b8d5-4b856bfcf1ea" volumeName="kubernetes.io/configmap/0ce80856-f167-427e-b8d5-4b856bfcf1ea-cni-binary-copy" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507833    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11efd367-1b70-4cb5-93c5-2dda7b9bb010" volumeName="kubernetes.io/projected/11efd367-1b70-4cb5-93c5-2dda7b9bb010-kube-api-access-nxxqd" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507846    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b8c537b-69a3-495b-a99b-afcbdc20ddf6" volumeName="kubernetes.io/configmap/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-env-overrides" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507860    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b8c537b-69a3-495b-a99b-afcbdc20ddf6" volumeName="kubernetes.io/configmap/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507873    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b8c537b-69a3-495b-a99b-afcbdc20ddf6" volumeName="kubernetes.io/projected/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-kube-api-access-456qc" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507887    4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e71e09c-79f2-42d0-a2ee-e1535cf3df8e" volumeName="kubernetes.io/configmap/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e-serviceca" seLinuxMountContext=""
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507899    4294 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.507906    4294 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.508466    4294 factory.go:153] Registering CRI-O factory
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.508482    4294 factory.go:223] Registration of the crio container factory successfully
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.508540    4294 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.508560    4294 factory.go:103] Registering Raw factory
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.508577    4294 manager.go:1199] Started watching for new ooms in manager
Nov 02 22:52:40.509146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.508923    4294 manager.go:319] Starting recovery of all containers
Nov 02 22:52:40.538521 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.538457    4294 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 22:52:40.539803 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.539779    4294 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 22:52:40.539803 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.539805    4294 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 22:52:40.540218 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.539825    4294 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 22:52:40.540218 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:52:40.539864    4294 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 22:52:40.560401 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.560364    4294 generic.go:356] "Generic (PLEG): container finished" podID="b1bd54f8-715f-428f-998b-65c6e7f8830e" containerID="ab9f13bcd95a1f5992b6542e4ee057ca23fe756e89afe5762e0f65685141f7c1" exitCode=0
Nov 02 22:52:40.560401 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.560402    4294 generic.go:356] "Generic (PLEG): container finished" podID="b1bd54f8-715f-428f-998b-65c6e7f8830e" containerID="a9da7d64f1eec3ead3ffe9d980cc570574a5ffdfe9c47a6ec341995417572a8d" exitCode=0
Nov 02 22:52:40.560609 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.560413    4294 generic.go:356] "Generic (PLEG): container finished" podID="b1bd54f8-715f-428f-998b-65c6e7f8830e" containerID="e90e6debd6f3424cf9933d059a3ee9039bcf83355e74fa8fa3e6835dfe36bf49" exitCode=0
Nov 02 22:52:40.560609 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.560422    4294 generic.go:356] "Generic (PLEG): container finished" podID="b1bd54f8-715f-428f-998b-65c6e7f8830e" containerID="2068d7388dc78cd4cfe2998de11538078bbc2abb7c535d2fd0ea01120bd12f44" exitCode=0
Nov 02 22:52:40.560609 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.560431    4294 generic.go:356] "Generic (PLEG): container finished" podID="b1bd54f8-715f-428f-998b-65c6e7f8830e" containerID="f34c6059a516d1cca0f150efe569ef21f4ecfbaac50fbdf29d5128008365612d" exitCode=0
Nov 02 22:52:40.560609 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.560439    4294 generic.go:356] "Generic (PLEG): container finished" podID="b1bd54f8-715f-428f-998b-65c6e7f8830e" containerID="e508006d6bd0317f5109bc1293f57bf6883d22bad6b14486b7e69ca7e0f03648" exitCode=0
Nov 02 22:52:40.566219 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.566195    4294 manager.go:324] Recovery completed
Nov 02 22:52:40.567721 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.567694    4294 generic.go:356] "Generic (PLEG): container finished" podID="a401e1afc8eeebc2962b0eea90d39ec6" containerID="200213f77723760ed486da6b92f7370c9a6272689532a2e44bfc4f4e2ee2bd3c" exitCode=0
Nov 02 22:52:40.579096 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.579073    4294 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 22:52:40.580397 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.580378    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 22:52:40.580506 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.580414    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 22:52:40.580506 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.580429    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 22:52:40.581659 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.581642    4294 cpu_manager.go:226] "Starting" policy="none"
Nov 02 22:52:40.581659 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.581655    4294 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 22:52:40.581659 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.581658    4294 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 22:52:40.581816 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.581670    4294 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 22:52:40.581816 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.581790    4294 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 22:52:40.581816 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.581801    4294 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 22:52:40.581907 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.581823    4294 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 22:52:40.581907 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.581829    4294 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 22:52:40.581907 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.581834    4294 policy_none.go:50] "Start"
Nov 02 22:52:40.581907 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.581842    4294 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 22:52:40.581907 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.581851    4294 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 22:52:40.582066 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.581997    4294 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 22:52:40.582066 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.582009    4294 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 22:52:40.582066 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.582017    4294 policy_none.go:44] "Start"
Nov 02 22:52:40.582683 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.582667    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 22:52:40.582754 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.582700    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 22:52:40.582754 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.582714    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 22:52:40.585872 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.585846    4294 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 22:52:40.588293 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.588278    4294 manager.go:341] "Starting Device Plugin manager"
Nov 02 22:52:40.588381 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:52:40.588301    4294 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 22:52:40.588381 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.588310    4294 server.go:87] "Starting device plugin registration server"
Nov 02 22:52:40.588630 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.588616    4294 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 22:52:40.588690 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.588631    4294 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 22:52:40.588739 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.588700    4294 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 22:52:40.588831 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.588795    4294 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 22:52:40.588884 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.588845    4294 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 22:52:40.588884 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.588855    4294 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 22:52:40.588997 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.588903    4294 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T22:52:40.588830919Z","UUID":"1db2c41b-917b-4e30-ba43-e771e08ed6f2","Handler":null,"Name":"","Endpoint":""}
Nov 02 22:52:40.590318 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.590299    4294 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 22:52:40.590413 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:52:40.590312    4294 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 22:52:40.590413 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.590335    4294 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 22:52:40.590413 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:52:40.590349    4294 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-234.us-west-2.compute.internal\" not found"
Nov 02 22:52:40.640870 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.640276    4294 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal"]
Nov 02 22:52:40.641068 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.641026    4294 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 22:52:40.642295 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.642275    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 22:52:40.642427 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.642307    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 22:52:40.642427 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.642319    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 22:52:40.642540 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.642529    4294 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 22:52:40.642687 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.642663    4294 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 22:52:40.643495 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.643473    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 22:52:40.643495 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.643492    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 22:52:40.643613 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.643505    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 22:52:40.643613 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.643512    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 22:52:40.643613 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.643517    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 22:52:40.643613 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.643523    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 22:52:40.643738 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:52:40.643672    4294 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-234.us-west-2.compute.internal\" not found" node="ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:40.643782 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.643741    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal" event={"ID":"990ba2ba1d2d380f95d805ba47d10a19","Type":"ContainerStarted","Data":"9a0e34bc268dc13e29f9b726d2535f600c650d683f9221bbdb4a6a22be2df0dc"}
Nov 02 22:52:40.643782 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.643774    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal" event={"ID":"990ba2ba1d2d380f95d805ba47d10a19","Type":"ContainerStarted","Data":"2dcd50348bcf9c60f8141439ad5bc51f5e57c17db613473548270e0ebaa46be2"}
Nov 02 22:52:40.643847 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.643788    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal" event={"ID":"a401e1afc8eeebc2962b0eea90d39ec6","Type":"ContainerStarted","Data":"983c1dbdad59fb51d40e5ce48b0cbbfeae86ac7724b92ecd31056b1d1bcdf682"}
Nov 02 22:52:40.643847 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.643810    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal" event={"ID":"a401e1afc8eeebc2962b0eea90d39ec6","Type":"ContainerDied","Data":"200213f77723760ed486da6b92f7370c9a6272689532a2e44bfc4f4e2ee2bd3c"}
Nov 02 22:52:40.643847 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.643822    4294 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 22:52:40.643847 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.643823    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal" event={"ID":"a401e1afc8eeebc2962b0eea90d39ec6","Type":"ContainerStarted","Data":"87fe95f3eb83273324f4440483aac39cb7cf35a79c053349b628fb609ed9c00d"}
Nov 02 22:52:40.644698 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.644681    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 22:52:40.644762 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.644708    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 22:52:40.644762 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.644718    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 22:52:40.644849 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:52:40.644837    4294 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-234.us-west-2.compute.internal\" not found" node="ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:40.688835 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.688784    4294 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 22:52:40.690135 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.690107    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 22:52:40.690237 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.690148    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 22:52:40.690237 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.690163    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 22:52:40.690237 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.690188    4294 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:40.737834 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.737749    4294 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 22:52:40.737834 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.737819    4294 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 22:52:40.738053 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.737924    4294 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 22:52:40.738053 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.737950    4294 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 22:52:40.785587 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.785547    4294 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 22:52:40.807184 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.807158    4294 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:40.834872 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.834845    4294 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:40.834958 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.834917    4294 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:40.835540 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.835523    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeNotReady"
Nov 02 22:52:40.835605 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.835554    4294 setters.go:546] "Node became not ready" node="ip-10-0-139-234.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T22:52:40Z","lastTransitionTime":"2026-11-02T22:52:40Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 02 22:52:40.838688 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.838669    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/990ba2ba1d2d380f95d805ba47d10a19-config\") pod \"kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal\" (UID: \"990ba2ba1d2d380f95d805ba47d10a19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:40.838737 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.838697    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a401e1afc8eeebc2962b0eea90d39ec6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal\" (UID: \"a401e1afc8eeebc2962b0eea90d39ec6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:40.838737 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.838713    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a401e1afc8eeebc2962b0eea90d39ec6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal\" (UID: \"a401e1afc8eeebc2962b0eea90d39ec6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:40.845949 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.845932    4294 nodeinfomanager.go:471] existing CSINode "ip-10-0-139-234.us-west-2.compute.internal" is owned by different node (oldNodeID="3b30dc5c-9493-4126-ad23-99c45252a985", newNodeID=""), cleaning up...
Nov 02 22:52:40.863342 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.863322    4294 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 22:52:40.863389 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:52:40.863367    4294 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:40.863389 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.863382    4294 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:40.919917 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.919885    4294 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 22:52:40.920065 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:52:40.919935    4294 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:40.939101 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.939073    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/990ba2ba1d2d380f95d805ba47d10a19-config\") pod \"kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal\" (UID: \"990ba2ba1d2d380f95d805ba47d10a19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:40.939174 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.939110    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a401e1afc8eeebc2962b0eea90d39ec6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal\" (UID: \"a401e1afc8eeebc2962b0eea90d39ec6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:40.939174 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.939134    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a401e1afc8eeebc2962b0eea90d39ec6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal\" (UID: \"a401e1afc8eeebc2962b0eea90d39ec6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:40.939267 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.939178    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/990ba2ba1d2d380f95d805ba47d10a19-config\") pod \"kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal\" (UID: \"990ba2ba1d2d380f95d805ba47d10a19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:40.939267 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.939183    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a401e1afc8eeebc2962b0eea90d39ec6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal\" (UID: \"a401e1afc8eeebc2962b0eea90d39ec6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:40.939267 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:40.939187    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a401e1afc8eeebc2962b0eea90d39ec6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal\" (UID: \"a401e1afc8eeebc2962b0eea90d39ec6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:41.493635 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.493600    4294 apiserver.go:52] "Watching apiserver"
Nov 02 22:52:41.575915 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.575886    4294 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:41.576073 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.575996    4294 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:41.592120 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.592100    4294 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 22:52:41.592458 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.592440    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-295c4","kube-system/global-pull-secret-syncer-hlfzd","kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl","openshift-image-registry/node-ca-hfl65","openshift-multus/multus-q5cp2","openshift-multus/network-metrics-daemon-6gn7z","openshift-dns/node-resolver-r2vkn","openshift-multus/multus-additional-cni-plugins-kbhxn","kube-system/konnectivity-agent-6zx4w","openshift-cluster-node-tuning-operator/tuned-smds5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-qhbsn","openshift-network-operator/iptables-alerter-p5hzg"]
Nov 02 22:52:41.596411 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.596383    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:41.607494 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.607476    4294 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 22:52:41.631962 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.631939    4294 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 22:52:41.632068 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:52:41.632006    4294 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:41.632279 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.632259    4294 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 22:52:41.632337 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:52:41.632324    4294 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-2.compute.internal"
Nov 02 22:52:41.643634 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.643607    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-ovn-node-metrics-cert\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.643746 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.643649    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-lib-modules\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.643746 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.643675    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-kubelet\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.643746 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.643699    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-run-ovn-kubernetes\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.643884 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.643809    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92845990-417f-4c8f-aa67-ea9961516288-hosts-file\") pod \"node-resolver-r2vkn\" (UID: \"92845990-417f-4c8f-aa67-ea9961516288\") " pod="openshift-dns/node-resolver-r2vkn"
Nov 02 22:52:41.643884 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.643861    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4nhs\" (UniqueName: \"kubernetes.io/projected/92845990-417f-4c8f-aa67-ea9961516288-kube-api-access-c4nhs\") pod \"node-resolver-r2vkn\" (UID: \"92845990-417f-4c8f-aa67-ea9961516288\") " pod="openshift-dns/node-resolver-r2vkn"
Nov 02 22:52:41.644057 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.643939    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1bd54f8-715f-428f-998b-65c6e7f8830e-system-cni-dir\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:41.644057 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.643967    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-run-netns\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.644057 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644023    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-var-lib-cni-bin\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.644057 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644048    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-sysconfig\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.644274 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644071    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-kubernetes\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.644274 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644122    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-sys\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.644274 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644165    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1bd54f8-715f-428f-998b-65c6e7f8830e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:41.644274 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644227    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e-serviceca\") pod \"node-ca-hfl65\" (UID: \"8e71e09c-79f2-42d0-a2ee-e1535cf3df8e\") " pod="openshift-image-registry/node-ca-hfl65"
Nov 02 22:52:41.644274 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644251    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-node-log\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.644565 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644293    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-log-socket\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.644565 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644332    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-456qc\" (UniqueName: \"kubernetes.io/projected/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-kube-api-access-456qc\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.644565 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644358    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 22:52:41.644565 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644358    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxxqd\" (UniqueName: \"kubernetes.io/projected/11efd367-1b70-4cb5-93c5-2dda7b9bb010-kube-api-access-nxxqd\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:41.644565 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644405    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9qmn\" (UniqueName: \"kubernetes.io/projected/e37c6a5d-1170-4f2d-ae18-2580da36f53b-kube-api-access-j9qmn\") pod \"iptables-alerter-p5hzg\" (UID: \"e37c6a5d-1170-4f2d-ae18-2580da36f53b\") " pod="openshift-network-operator/iptables-alerter-p5hzg"
Nov 02 22:52:41.644565 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644430    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-ovnkube-script-lib\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.644565 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644445    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 22:52:41.644565 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644455    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-slash\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.644565 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644478    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-run-ovn\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.644565 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644502    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-registration-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:41.644565 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644527    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1bd54f8-715f-428f-998b-65c6e7f8830e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:41.644565 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644548    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 22:52:41.644565 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644552    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0dfa862-aaf8-4ff1-820e-a610a4d41ba8-agent-certs\") pod \"konnectivity-agent-6zx4w\" (UID: \"a0dfa862-aaf8-4ff1-820e-a610a4d41ba8\") " pod="kube-system/konnectivity-agent-6zx4w"
Nov 02 22:52:41.644565 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644478    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 22:52:41.644565 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644561    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644581    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644623    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-etc-kubernetes\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644675    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644684    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18dcacc0-751d-4f2e-8101-59a1da48fab2-tmp\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644692    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644736    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-run-openvswitch\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644765    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs\") pod \"network-metrics-daemon-6gn7z\" (UID: \"4b1cf92b-28eb-4894-96ed-b7eba4a781d2\") " pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644790    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-system-cni-dir\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644823    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644832    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18dcacc0-751d-4f2e-8101-59a1da48fab2-tmp\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644843    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ce80856-f167-427e-b8d5-4b856bfcf1ea-cni-binary-copy\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644892    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644910    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-var-lib-cni-multus\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.644943    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-systemd\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645000    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-systemd-units\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645047    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-run-netns\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645072    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-sys-fs\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645094    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sdz5j\""
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645101    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hl8b\" (UniqueName: \"kubernetes.io/projected/b1bd54f8-715f-428f-998b-65c6e7f8830e-kube-api-access-5hl8b\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645128    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0dfa862-aaf8-4ff1-820e-a610a4d41ba8-konnectivity-ca\") pod \"konnectivity-agent-6zx4w\" (UID: \"a0dfa862-aaf8-4ff1-820e-a610a4d41ba8\") " pod="kube-system/konnectivity-agent-6zx4w"
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645144    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9qgnp\""
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645145    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645174    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645174    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-v5hvl\""
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645203    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c9lzr\""
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645247    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-n2bfg\""
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645269    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645307    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645325    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645325    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-modprobe-d\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.645360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645356    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645389    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-run\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645414    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645415    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-var-lib-openvswitch\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645441    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-cni-bin\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645478    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-socket-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645497    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxcv\" (UniqueName: \"kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv\") pod \"network-check-target-qhbsn\" (UID: \"8161cb48-56bb-455b-bee6-cd3d06607109\") " pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645514    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-daemon-config\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645557    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-sysctl-conf\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645594    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-device-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645693    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-etc-selinux\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645715    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-run-k8s-cni-cncf-io\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645734    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-var-lib-kubelet\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645767    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-etc-openvswitch\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645794    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645834    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-daemon-config\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645842    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-ovnkube-config\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645880    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-env-overrides\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645929    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5kxql\""
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.645942    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646014    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-hostroot\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646045    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e37c6a5d-1170-4f2d-ae18-2580da36f53b-iptables-alerter-script\") pod \"iptables-alerter-p5hzg\" (UID: \"e37c6a5d-1170-4f2d-ae18-2580da36f53b\") " pod="openshift-network-operator/iptables-alerter-p5hzg"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646073    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-socket-dir-parent\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646088    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-env-overrides\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646130    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92845990-417f-4c8f-aa67-ea9961516288-tmp-dir\") pod \"node-resolver-r2vkn\" (UID: \"92845990-417f-4c8f-aa67-ea9961516288\") " pod="openshift-dns/node-resolver-r2vkn"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646133    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-ovnkube-config\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646170    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-cnibin\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646207    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92845990-417f-4c8f-aa67-ea9961516288-tmp-dir\") pod \"node-resolver-r2vkn\" (UID: \"92845990-417f-4c8f-aa67-ea9961516288\") " pod="openshift-dns/node-resolver-r2vkn"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646222    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-tuned\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646256    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646265    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a474267-46b1-4759-9018-54960f751878-original-pull-secret\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646288    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1bd54f8-715f-428f-998b-65c6e7f8830e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646299    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646324    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-tuned\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646333    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646339    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9m47\" (UniqueName: \"kubernetes.io/projected/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e-kube-api-access-x9m47\") pod \"node-ca-hfl65\" (UID: \"8e71e09c-79f2-42d0-a2ee-e1535cf3df8e\") " pod="openshift-image-registry/node-ca-hfl65"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646372    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646391    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-cni-dir\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646352    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rxqrv\""
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646408    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-conf-dir\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646435    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-run-systemd\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646460    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1bd54f8-715f-428f-998b-65c6e7f8830e-cni-binary-copy\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646526    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a474267-46b1-4759-9018-54960f751878-kubelet-config\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646556    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a474267-46b1-4759-9018-54960f751878-dbus\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646580    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-run-multus-certs\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646618    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6hww\" (UniqueName: \"kubernetes.io/projected/0ce80856-f167-427e-b8d5-4b856bfcf1ea-kube-api-access-r6hww\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646661    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9zml\" (UniqueName: \"kubernetes.io/projected/18dcacc0-751d-4f2e-8101-59a1da48fab2-kube-api-access-g9zml\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646664    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1bd54f8-715f-428f-998b-65c6e7f8830e-cni-binary-copy\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646702    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25fvj\" (UniqueName: \"kubernetes.io/projected/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-kube-api-access-25fvj\") pod \"network-metrics-daemon-6gn7z\" (UID: \"4b1cf92b-28eb-4894-96ed-b7eba4a781d2\") " pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:41.646820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646827    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e37c6a5d-1170-4f2d-ae18-2580da36f53b-host-slash\") pod \"iptables-alerter-p5hzg\" (UID: \"e37c6a5d-1170-4f2d-ae18-2580da36f53b\") " pod="openshift-network-operator/iptables-alerter-p5hzg"
Nov 02 22:52:41.648458 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646877    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 22:52:41.648458 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646901    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-92nqq\""
Nov 02 22:52:41.648458 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646912    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 22:52:41.648458 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646942    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-var-lib-kubelet\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.648458 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646958    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-host\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.648458 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.646975    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1bd54f8-715f-428f-998b-65c6e7f8830e-cnibin\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:41.648458 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.647028    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e-host\") pod \"node-ca-hfl65\" (UID: \"8e71e09c-79f2-42d0-a2ee-e1535cf3df8e\") " pod="openshift-image-registry/node-ca-hfl65"
Nov 02 22:52:41.648458 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.647052    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-sysctl-d\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.648458 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.647068    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-cni-netd\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.648458 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.647197    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-crtkq\""
Nov 02 22:52:41.648458 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.647428    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 22:52:41.648458 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.647433    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 22:52:41.648458 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.647437    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 22:52:41.648458 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.647459    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 22:52:41.648458 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.647462    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9p98l\""
Nov 02 22:52:41.654860 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.654842    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1bd54f8-715f-428f-998b-65c6e7f8830e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:41.654930 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.654896    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0dfa862-aaf8-4ff1-820e-a610a4d41ba8-agent-certs\") pod \"konnectivity-agent-6zx4w\" (UID: \"a0dfa862-aaf8-4ff1-820e-a610a4d41ba8\") " pod="kube-system/konnectivity-agent-6zx4w"
Nov 02 22:52:41.654930 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.654915    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-ovnkube-script-lib\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.655083 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.655069    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ce80856-f167-427e-b8d5-4b856bfcf1ea-cni-binary-copy\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.655143 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.655126    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-ovn-node-metrics-cert\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.655397 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.655382    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e-serviceca\") pod \"node-ca-hfl65\" (UID: \"8e71e09c-79f2-42d0-a2ee-e1535cf3df8e\") " pod="openshift-image-registry/node-ca-hfl65"
Nov 02 22:52:41.655546 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.655532    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0dfa862-aaf8-4ff1-820e-a610a4d41ba8-konnectivity-ca\") pod \"konnectivity-agent-6zx4w\" (UID: \"a0dfa862-aaf8-4ff1-820e-a610a4d41ba8\") " pod="kube-system/konnectivity-agent-6zx4w"
Nov 02 22:52:41.657131 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.657116    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1bd54f8-715f-428f-998b-65c6e7f8830e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:41.679764 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.679744    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2rjj4\""
Nov 02 22:52:41.680264 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.680251    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-77brg\""
Nov 02 22:52:41.680299 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.680288    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 22:52:41.690065 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.690049    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 22:52:41.690666 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.690652    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 22:52:41.690754 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.690670    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 22:52:41.690754 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.690701    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 22:52:41.696990 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.696967    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e37c6a5d-1170-4f2d-ae18-2580da36f53b-iptables-alerter-script\") pod \"iptables-alerter-p5hzg\" (UID: \"e37c6a5d-1170-4f2d-ae18-2580da36f53b\") " pod="openshift-network-operator/iptables-alerter-p5hzg"
Nov 02 22:52:41.699958 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.699939    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-456qc\" (UniqueName: \"kubernetes.io/projected/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-kube-api-access-456qc\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.700150 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.700127    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4nhs\" (UniqueName: \"kubernetes.io/projected/92845990-417f-4c8f-aa67-ea9961516288-kube-api-access-c4nhs\") pod \"node-resolver-r2vkn\" (UID: \"92845990-417f-4c8f-aa67-ea9961516288\") " pod="openshift-dns/node-resolver-r2vkn"
Nov 02 22:52:41.700264 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.700250    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxxqd\" (UniqueName: \"kubernetes.io/projected/11efd367-1b70-4cb5-93c5-2dda7b9bb010-kube-api-access-nxxqd\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:41.701342 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.701322    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hl8b\" (UniqueName: \"kubernetes.io/projected/b1bd54f8-715f-428f-998b-65c6e7f8830e-kube-api-access-5hl8b\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:41.701403 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.701328    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9qmn\" (UniqueName: \"kubernetes.io/projected/e37c6a5d-1170-4f2d-ae18-2580da36f53b-kube-api-access-j9qmn\") pod \"iptables-alerter-p5hzg\" (UID: \"e37c6a5d-1170-4f2d-ae18-2580da36f53b\") " pod="openshift-network-operator/iptables-alerter-p5hzg"
Nov 02 22:52:41.701569 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.701553    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drxcv\" (UniqueName: \"kubernetes.io/projected/8161cb48-56bb-455b-bee6-cd3d06607109-kube-api-access-drxcv\") pod \"network-check-target-qhbsn\" (UID: \"8161cb48-56bb-455b-bee6-cd3d06607109\") " pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:41.704343 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.704320    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9m47\" (UniqueName: \"kubernetes.io/projected/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e-kube-api-access-x9m47\") pod \"node-ca-hfl65\" (UID: \"8e71e09c-79f2-42d0-a2ee-e1535cf3df8e\") " pod="openshift-image-registry/node-ca-hfl65"
Nov 02 22:52:41.704471 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.704351    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6hww\" (UniqueName: \"kubernetes.io/projected/0ce80856-f167-427e-b8d5-4b856bfcf1ea-kube-api-access-r6hww\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.704471 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.704400    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9zml\" (UniqueName: \"kubernetes.io/projected/18dcacc0-751d-4f2e-8101-59a1da48fab2-kube-api-access-g9zml\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.704748 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.704726    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25fvj\" (UniqueName: \"kubernetes.io/projected/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-kube-api-access-25fvj\") pod \"network-metrics-daemon-6gn7z\" (UID: \"4b1cf92b-28eb-4894-96ed-b7eba4a781d2\") " pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:41.747941 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.747873    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-cnibin\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.747941 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.747904    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:41.747941 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.747936    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-cni-dir\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.748138 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.747952    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-conf-dir\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.748138 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.747974    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-run-systemd\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.748138 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748033    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-run-systemd\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.748138 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748035    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-conf-dir\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.748138 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748035    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:41.748138 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748057    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a474267-46b1-4759-9018-54960f751878-kubelet-config\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:41.748138 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748091    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a474267-46b1-4759-9018-54960f751878-kubelet-config\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:41.748138 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748113    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a474267-46b1-4759-9018-54960f751878-dbus\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:41.748138 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748124    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-cni-dir\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.748138 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748137    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-run-multus-certs\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.748138 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748115    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-cnibin\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748196    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e37c6a5d-1170-4f2d-ae18-2580da36f53b-host-slash\") pod \"iptables-alerter-p5hzg\" (UID: \"e37c6a5d-1170-4f2d-ae18-2580da36f53b\") " pod="openshift-network-operator/iptables-alerter-p5hzg"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748208    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-run-multus-certs\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748215    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a474267-46b1-4759-9018-54960f751878-dbus\") pod \"global-pull-secret-syncer-hlfzd\" (UID: \"3a474267-46b1-4759-9018-54960f751878\") " pod="kube-system/global-pull-secret-syncer-hlfzd"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748226    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-var-lib-kubelet\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748242    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e37c6a5d-1170-4f2d-ae18-2580da36f53b-host-slash\") pod \"iptables-alerter-p5hzg\" (UID: \"e37c6a5d-1170-4f2d-ae18-2580da36f53b\") " pod="openshift-network-operator/iptables-alerter-p5hzg"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748270    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-host\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748297    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1bd54f8-715f-428f-998b-65c6e7f8830e-cnibin\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748308    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-var-lib-kubelet\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748348    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-host\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748375    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e-host\") pod \"node-ca-hfl65\" (UID: \"8e71e09c-79f2-42d0-a2ee-e1535cf3df8e\") " pod="openshift-image-registry/node-ca-hfl65"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748394    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1bd54f8-715f-428f-998b-65c6e7f8830e-cnibin\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748400    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-sysctl-d\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748424    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8e71e09c-79f2-42d0-a2ee-e1535cf3df8e-host\") pod \"node-ca-hfl65\" (UID: \"8e71e09c-79f2-42d0-a2ee-e1535cf3df8e\") " pod="openshift-image-registry/node-ca-hfl65"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748424    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-cni-netd\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748469    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-lib-modules\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748491    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-kubelet\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748504    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-sysctl-d\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748512    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-cni-netd\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748514    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-run-ovn-kubernetes\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748549    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-kubelet\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748569    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92845990-417f-4c8f-aa67-ea9961516288-hosts-file\") pod \"node-resolver-r2vkn\" (UID: \"92845990-417f-4c8f-aa67-ea9961516288\") " pod="openshift-dns/node-resolver-r2vkn"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748594    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-lib-modules\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748603    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1bd54f8-715f-428f-998b-65c6e7f8830e-system-cni-dir\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748618    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/92845990-417f-4c8f-aa67-ea9961516288-hosts-file\") pod \"node-resolver-r2vkn\" (UID: \"92845990-417f-4c8f-aa67-ea9961516288\") " pod="openshift-dns/node-resolver-r2vkn"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748628    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-run-netns\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748607    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-run-ovn-kubernetes\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748635    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1bd54f8-715f-428f-998b-65c6e7f8830e-system-cni-dir\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748654    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-run-netns\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.748651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748665    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-var-lib-cni-bin\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748687    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-sysconfig\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748699    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-var-lib-cni-bin\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748706    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-kubernetes\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748747    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-sysconfig\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748751    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-sys\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748785    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-kubernetes\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748795    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1bd54f8-715f-428f-998b-65c6e7f8830e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748811    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-sys\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748854    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1bd54f8-715f-428f-998b-65c6e7f8830e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kbhxn\" (UID: \"b1bd54f8-715f-428f-998b-65c6e7f8830e\") " pod="openshift-multus/multus-additional-cni-plugins-kbhxn"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748887    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-node-log\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748911    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-node-log\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749041    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-log-socket\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.748931    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-log-socket\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749370    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-slash\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749384    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-run-ovn\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749401    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-registration-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749417    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-etc-kubernetes\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749431    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-run-openvswitch\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749451    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs\") pod \"network-metrics-daemon-6gn7z\" (UID: \"4b1cf92b-28eb-4894-96ed-b7eba4a781d2\") " pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749458    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-run-ovn\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749473    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-system-cni-dir\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749480    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-etc-kubernetes\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749500    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-var-lib-cni-multus\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749504    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-slash\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749500    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-run-openvswitch\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749576    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-registration-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749581    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-systemd\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749582    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-var-lib-cni-multus\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749584    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-system-cni-dir\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749628    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-systemd\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749625    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-systemd-units\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749682    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-systemd-units\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749712    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-run-netns\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749750    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-run-netns\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.749770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749779    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-sys-fs\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749824    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-modprobe-d\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749859    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-sys-fs\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749858    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-run\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749915    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-modprobe-d\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749920    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-var-lib-openvswitch\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749949    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-cni-bin\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749954    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-run\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.749965    4294 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750026    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-socket-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750046    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-cni-bin\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750051    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-sysctl-conf\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750075    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-socket-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750026    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-var-lib-openvswitch\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750095    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-device-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750142    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-etc-selinux\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750175    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/18dcacc0-751d-4f2e-8101-59a1da48fab2-etc-sysctl-conf\") pod \"tuned-smds5\" (UID: \"18dcacc0-751d-4f2e-8101-59a1da48fab2\") " pod="openshift-cluster-node-tuning-operator/tuned-smds5"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750174    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-run-k8s-cni-cncf-io\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750178    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-device-dir\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750216    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-run-k8s-cni-cncf-io\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750216    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/11efd367-1b70-4cb5-93c5-2dda7b9bb010-etc-selinux\") pod \"aws-ebs-csi-driver-node-j94hl\" (UID: \"11efd367-1b70-4cb5-93c5-2dda7b9bb010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j94hl"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750216    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-var-lib-kubelet\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750244    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-host-var-lib-kubelet\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750271    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-etc-openvswitch\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750304    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-etc-openvswitch\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750311    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750374    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-hostroot\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750389    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-socket-dir-parent\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750388    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b8c537b-69a3-495b-a99b-afcbdc20ddf6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-295c4\" (UID: \"5b8c537b-69a3-495b-a99b-afcbdc20ddf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750423    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-hostroot\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.750901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.750454    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ce80856-f167-427e-b8d5-4b856bfcf1ea-multus-socket-dir-parent\") pod \"multus-q5cp2\" (UID: \"0ce80856-f167-427e-b8d5-4b856bfcf1ea\") " pod="openshift-multus/multus-q5cp2"
Nov 02 22:52:41.752181 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.752156    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1cf92b-28eb-4894-96ed-b7eba4a781d2-metrics-certs\") pod \"network-metrics-daemon-6gn7z\" (UID: \"4b1cf92b-28eb-4894-96ed-b7eba4a781d2\") " pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:41.863837 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.863758    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:41.897490 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:41.897453    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6gn7z"
Nov 02 22:52:42.066919 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:42.066832    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:42.254781 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:42.254700    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6gn7z"]
Nov 02 22:52:42.257241 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:42.257202    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1cf92b_28eb_4894_96ed_b7eba4a781d2.slice/crio-d6cdd5afe7c1d240053ef6ffc52adb887135f95397cd1b309fc5be989a01f52d WatchSource:0}: Error finding container d6cdd5afe7c1d240053ef6ffc52adb887135f95397cd1b309fc5be989a01f52d: Status 404 returned error can't find the container with id d6cdd5afe7c1d240053ef6ffc52adb887135f95397cd1b309fc5be989a01f52d
Nov 02 22:52:42.259546 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:42.259523    4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 22:52:42.269574 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:42.269549    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:42.462999 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:42.462904    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:42.579370 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:42.579332    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6gn7z" event={"ID":"4b1cf92b-28eb-4894-96ed-b7eba4a781d2","Type":"ContainerStarted","Data":"d6cdd5afe7c1d240053ef6ffc52adb887135f95397cd1b309fc5be989a01f52d"}
Nov 02 22:52:42.670436 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:42.670400    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:44.585958 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:44.585923    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6gn7z" event={"ID":"4b1cf92b-28eb-4894-96ed-b7eba4a781d2","Type":"ContainerStarted","Data":"d8f7c7a79af658bd58af3e411cff66ccd17e4b463ae6a3e0fe2a784662f00a9b"}
Nov 02 22:52:44.586376 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:44.586004    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6gn7z" event={"ID":"4b1cf92b-28eb-4894-96ed-b7eba4a781d2","Type":"ContainerStarted","Data":"fcc9b8e234ab2fa93096dd53fd52301b36a9ac0c6261a96ac8fdfbd61dacc484"}
Nov 02 22:52:46.616370 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:46.616330    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:46.767581 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:46.767548    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:46.930203 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:46.930123    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:47.094681 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:47.094650    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:47.243803 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:47.243764    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:47.536380 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:47.536299    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:48.765254 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:48.765226    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:48.967958 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:48.967922    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:49.089163 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:49.089091    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6zx4w"
Nov 02 22:52:49.331841 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:49.331806    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:49.438631 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:49.438545    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:52:49.494996 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:49.494947    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:49.680074 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:49.680040    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:49.832969 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:49.832933    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:50.106634 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:50.106548    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:50.919173 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:50.919146    4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-2.compute.internal" event="NodeReady"
Nov 02 22:52:51.006756 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.006725    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r4lgf"]
Nov 02 22:52:51.103511 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.103479    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ngvq8"]
Nov 02 22:52:51.103665 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.103613    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r4lgf"
Nov 02 22:52:51.117447 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.117430    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r4lgf"]
Nov 02 22:52:51.117550 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.117466    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ngvq8"]
Nov 02 22:52:51.117550 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.117491    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tr9pt"]
Nov 02 22:52:51.117630 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.117614    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ngvq8"
Nov 02 22:52:51.131657 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.131638    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tr9pt"
Nov 02 22:52:51.153589 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.153568    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 22:52:51.153589 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.153579    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 22:52:51.153711 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.153579    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t6gjv\""
Nov 02 22:52:51.164570 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.164550    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tr9pt"]
Nov 02 22:52:51.205158 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.205104    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19ca5d3a-9921-4eec-bb4b-cb8166fb85d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngvq8\" (UID: \"19ca5d3a-9921-4eec-bb4b-cb8166fb85d5\") " pod="openshift-insights/insights-runtime-extractor-ngvq8"
Nov 02 22:52:51.205158 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.205132    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd2f118b-b69b-4b5e-a540-c28a023a7ab8-cert\") pod \"ingress-canary-tr9pt\" (UID: \"fd2f118b-b69b-4b5e-a540-c28a023a7ab8\") " pod="openshift-ingress-canary/ingress-canary-tr9pt"
Nov 02 22:52:51.205158 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.205150    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkbv2\" (UniqueName: \"kubernetes.io/projected/fd2f118b-b69b-4b5e-a540-c28a023a7ab8-kube-api-access-lkbv2\") pod \"ingress-canary-tr9pt\" (UID: \"fd2f118b-b69b-4b5e-a540-c28a023a7ab8\") " pod="openshift-ingress-canary/ingress-canary-tr9pt"
Nov 02 22:52:51.205359 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.205166    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnxf5\" (UniqueName: \"kubernetes.io/projected/de1dc693-c309-4d7c-b8dd-e1e7a435e8a1-kube-api-access-vnxf5\") pod \"dns-default-r4lgf\" (UID: \"de1dc693-c309-4d7c-b8dd-e1e7a435e8a1\") " pod="openshift-dns/dns-default-r4lgf"
Nov 02 22:52:51.205359 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.205219    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19ca5d3a-9921-4eec-bb4b-cb8166fb85d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngvq8\" (UID: \"19ca5d3a-9921-4eec-bb4b-cb8166fb85d5\") " pod="openshift-insights/insights-runtime-extractor-ngvq8"
Nov 02 22:52:51.205359 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.205313    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de1dc693-c309-4d7c-b8dd-e1e7a435e8a1-tmp-dir\") pod \"dns-default-r4lgf\" (UID: \"de1dc693-c309-4d7c-b8dd-e1e7a435e8a1\") " pod="openshift-dns/dns-default-r4lgf"
Nov 02 22:52:51.205359 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.205337    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/19ca5d3a-9921-4eec-bb4b-cb8166fb85d5-data-volume\") pod \"insights-runtime-extractor-ngvq8\" (UID: \"19ca5d3a-9921-4eec-bb4b-cb8166fb85d5\") " pod="openshift-insights/insights-runtime-extractor-ngvq8"
Nov 02 22:52:51.205359 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.205352    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/19ca5d3a-9921-4eec-bb4b-cb8166fb85d5-crio-socket\") pod \"insights-runtime-extractor-ngvq8\" (UID: \"19ca5d3a-9921-4eec-bb4b-cb8166fb85d5\") " pod="openshift-insights/insights-runtime-extractor-ngvq8"
Nov 02 22:52:51.205541 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.205368    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fczz8\" (UniqueName: \"kubernetes.io/projected/19ca5d3a-9921-4eec-bb4b-cb8166fb85d5-kube-api-access-fczz8\") pod \"insights-runtime-extractor-ngvq8\" (UID: \"19ca5d3a-9921-4eec-bb4b-cb8166fb85d5\") " pod="openshift-insights/insights-runtime-extractor-ngvq8"
Nov 02 22:52:51.205541 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.205387    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de1dc693-c309-4d7c-b8dd-e1e7a435e8a1-metrics-tls\") pod \"dns-default-r4lgf\" (UID: \"de1dc693-c309-4d7c-b8dd-e1e7a435e8a1\") " pod="openshift-dns/dns-default-r4lgf"
Nov 02 22:52:51.205541 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.205479    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/de1dc693-c309-4d7c-b8dd-e1e7a435e8a1-config-volume\") pod \"dns-default-r4lgf\" (UID: \"de1dc693-c309-4d7c-b8dd-e1e7a435e8a1\") " pod="openshift-dns/dns-default-r4lgf"
Nov 02 22:52:51.255031 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.254832    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 22:52:51.255031 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.255004    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 22:52:51.255153 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.255109    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 22:52:51.255153 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.255137    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 22:52:51.255249 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.255175    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 22:52:51.255249 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.255237    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8zgv4\""
Nov 02 22:52:51.255348 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.255258    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vbflc\""
Nov 02 22:52:51.255348 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.255265    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 22:52:51.255348 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.255288    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 22:52:51.287785 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.287754    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:52:51.287954 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.287930    4294 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 22:52:51.306103 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.306080    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/19ca5d3a-9921-4eec-bb4b-cb8166fb85d5-crio-socket\") pod \"insights-runtime-extractor-ngvq8\" (UID: \"19ca5d3a-9921-4eec-bb4b-cb8166fb85d5\") " pod="openshift-insights/insights-runtime-extractor-ngvq8"
Nov 02 22:52:51.306233 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.306111    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fczz8\" (UniqueName: \"kubernetes.io/projected/19ca5d3a-9921-4eec-bb4b-cb8166fb85d5-kube-api-access-fczz8\") pod \"insights-runtime-extractor-ngvq8\" (UID: \"19ca5d3a-9921-4eec-bb4b-cb8166fb85d5\") " pod="openshift-insights/insights-runtime-extractor-ngvq8"
Nov 02 22:52:51.306233 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.306138    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de1dc693-c309-4d7c-b8dd-e1e7a435e8a1-metrics-tls\") pod \"dns-default-r4lgf\" (UID: \"de1dc693-c309-4d7c-b8dd-e1e7a435e8a1\") " pod="openshift-dns/dns-default-r4lgf"
Nov 02 22:52:51.306233 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.306170    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/de1dc693-c309-4d7c-b8dd-e1e7a435e8a1-config-volume\") pod \"dns-default-r4lgf\" (UID: \"de1dc693-c309-4d7c-b8dd-e1e7a435e8a1\") " pod="openshift-dns/dns-default-r4lgf"
Nov 02 22:52:51.306233 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.306203    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/19ca5d3a-9921-4eec-bb4b-cb8166fb85d5-crio-socket\") pod \"insights-runtime-extractor-ngvq8\" (UID: \"19ca5d3a-9921-4eec-bb4b-cb8166fb85d5\") " pod="openshift-insights/insights-runtime-extractor-ngvq8"
Nov 02 22:52:51.306233 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.306202    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19ca5d3a-9921-4eec-bb4b-cb8166fb85d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngvq8\" (UID: \"19ca5d3a-9921-4eec-bb4b-cb8166fb85d5\") " pod="openshift-insights/insights-runtime-extractor-ngvq8"
Nov 02 22:52:51.306480 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.306254    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd2f118b-b69b-4b5e-a540-c28a023a7ab8-cert\") pod \"ingress-canary-tr9pt\" (UID: \"fd2f118b-b69b-4b5e-a540-c28a023a7ab8\") " pod="openshift-ingress-canary/ingress-canary-tr9pt"
Nov 02 22:52:51.306480 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.306282    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkbv2\" (UniqueName: \"kubernetes.io/projected/fd2f118b-b69b-4b5e-a540-c28a023a7ab8-kube-api-access-lkbv2\") pod \"ingress-canary-tr9pt\" (UID: \"fd2f118b-b69b-4b5e-a540-c28a023a7ab8\") " pod="openshift-ingress-canary/ingress-canary-tr9pt"
Nov 02 22:52:51.306480 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.306305    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnxf5\" (UniqueName: \"kubernetes.io/projected/de1dc693-c309-4d7c-b8dd-e1e7a435e8a1-kube-api-access-vnxf5\") pod \"dns-default-r4lgf\" (UID: \"de1dc693-c309-4d7c-b8dd-e1e7a435e8a1\") " pod="openshift-dns/dns-default-r4lgf"
Nov 02 22:52:51.306480 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.306332    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19ca5d3a-9921-4eec-bb4b-cb8166fb85d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngvq8\" (UID: \"19ca5d3a-9921-4eec-bb4b-cb8166fb85d5\") " pod="openshift-insights/insights-runtime-extractor-ngvq8"
Nov 02 22:52:51.306480 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.306366    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de1dc693-c309-4d7c-b8dd-e1e7a435e8a1-tmp-dir\") pod \"dns-default-r4lgf\" (UID: \"de1dc693-c309-4d7c-b8dd-e1e7a435e8a1\") " pod="openshift-dns/dns-default-r4lgf"
Nov 02 22:52:51.306480 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.306403    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/19ca5d3a-9921-4eec-bb4b-cb8166fb85d5-data-volume\") pod \"insights-runtime-extractor-ngvq8\" (UID: \"19ca5d3a-9921-4eec-bb4b-cb8166fb85d5\") " pod="openshift-insights/insights-runtime-extractor-ngvq8"
Nov 02 22:52:51.306887 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.306859    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de1dc693-c309-4d7c-b8dd-e1e7a435e8a1-tmp-dir\") pod \"dns-default-r4lgf\" (UID: \"de1dc693-c309-4d7c-b8dd-e1e7a435e8a1\") " pod="openshift-dns/dns-default-r4lgf"
Nov 02 22:52:51.307042 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.307019    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/19ca5d3a-9921-4eec-bb4b-cb8166fb85d5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ngvq8\" (UID: \"19ca5d3a-9921-4eec-bb4b-cb8166fb85d5\") " pod="openshift-insights/insights-runtime-extractor-ngvq8"
Nov 02 22:52:51.307158 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.307129    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/19ca5d3a-9921-4eec-bb4b-cb8166fb85d5-data-volume\") pod \"insights-runtime-extractor-ngvq8\" (UID: \"19ca5d3a-9921-4eec-bb4b-cb8166fb85d5\") " pod="openshift-insights/insights-runtime-extractor-ngvq8"
Nov 02 22:52:51.308471 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.308448    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/de1dc693-c309-4d7c-b8dd-e1e7a435e8a1-config-volume\") pod \"dns-default-r4lgf\" (UID: \"de1dc693-c309-4d7c-b8dd-e1e7a435e8a1\") " pod="openshift-dns/dns-default-r4lgf"
Nov 02 22:52:51.308951 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.308933    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/19ca5d3a-9921-4eec-bb4b-cb8166fb85d5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ngvq8\" (UID: \"19ca5d3a-9921-4eec-bb4b-cb8166fb85d5\") " pod="openshift-insights/insights-runtime-extractor-ngvq8"
Nov 02 22:52:51.309232 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.309214    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd2f118b-b69b-4b5e-a540-c28a023a7ab8-cert\") pod \"ingress-canary-tr9pt\" (UID: \"fd2f118b-b69b-4b5e-a540-c28a023a7ab8\") " pod="openshift-ingress-canary/ingress-canary-tr9pt"
Nov 02 22:52:51.309360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.309343    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/de1dc693-c309-4d7c-b8dd-e1e7a435e8a1-metrics-tls\") pod \"dns-default-r4lgf\" (UID: \"de1dc693-c309-4d7c-b8dd-e1e7a435e8a1\") " pod="openshift-dns/dns-default-r4lgf"
Nov 02 22:52:51.360615 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.360580    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fczz8\" (UniqueName: \"kubernetes.io/projected/19ca5d3a-9921-4eec-bb4b-cb8166fb85d5-kube-api-access-fczz8\") pod \"insights-runtime-extractor-ngvq8\" (UID: \"19ca5d3a-9921-4eec-bb4b-cb8166fb85d5\") " pod="openshift-insights/insights-runtime-extractor-ngvq8"
Nov 02 22:52:51.360750 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.360654    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkbv2\" (UniqueName: \"kubernetes.io/projected/fd2f118b-b69b-4b5e-a540-c28a023a7ab8-kube-api-access-lkbv2\") pod \"ingress-canary-tr9pt\" (UID: \"fd2f118b-b69b-4b5e-a540-c28a023a7ab8\") " pod="openshift-ingress-canary/ingress-canary-tr9pt"
Nov 02 22:52:51.360750 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.360678    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnxf5\" (UniqueName: \"kubernetes.io/projected/de1dc693-c309-4d7c-b8dd-e1e7a435e8a1-kube-api-access-vnxf5\") pod \"dns-default-r4lgf\" (UID: \"de1dc693-c309-4d7c-b8dd-e1e7a435e8a1\") " pod="openshift-dns/dns-default-r4lgf"
Nov 02 22:52:51.438423 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.438395    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r4lgf"
Nov 02 22:52:51.482368 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.482337    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ngvq8"
Nov 02 22:52:51.498272 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.498239    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tr9pt"
Nov 02 22:52:51.812873 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.812846    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r4lgf"]
Nov 02 22:52:51.815153 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:51.815100    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde1dc693_c309_4d7c_b8dd_e1e7a435e8a1.slice/crio-3dccc45a78644ba30273f64b22fc01e3e6a95446895ddbd058a0ddfe43f5817c WatchSource:0}: Error finding container 3dccc45a78644ba30273f64b22fc01e3e6a95446895ddbd058a0ddfe43f5817c: Status 404 returned error can't find the container with id 3dccc45a78644ba30273f64b22fc01e3e6a95446895ddbd058a0ddfe43f5817c
Nov 02 22:52:51.906574 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.906541    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tr9pt"]
Nov 02 22:52:51.909794 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:51.909767    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2f118b_b69b_4b5e_a540_c28a023a7ab8.slice/crio-0b6814368f4216659a611800e1ee5e7220db78b5b8e63aa4279a696dc03d1ee3 WatchSource:0}: Error finding container 0b6814368f4216659a611800e1ee5e7220db78b5b8e63aa4279a696dc03d1ee3: Status 404 returned error can't find the container with id 0b6814368f4216659a611800e1ee5e7220db78b5b8e63aa4279a696dc03d1ee3
Nov 02 22:52:51.914831 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:51.914805    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ngvq8"]
Nov 02 22:52:51.917263 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:51.917244    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ca5d3a_9921_4eec_bb4b_cb8166fb85d5.slice/crio-e17e12ef3e94df994d3ead5ce7d5e5e00a0ccf69244068ba70abbcb8bcecf529 WatchSource:0}: Error finding container e17e12ef3e94df994d3ead5ce7d5e5e00a0ccf69244068ba70abbcb8bcecf529: Status 404 returned error can't find the container with id e17e12ef3e94df994d3ead5ce7d5e5e00a0ccf69244068ba70abbcb8bcecf529
Nov 02 22:52:52.603603 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:52.603565    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r4lgf" event={"ID":"de1dc693-c309-4d7c-b8dd-e1e7a435e8a1","Type":"ContainerStarted","Data":"3dccc45a78644ba30273f64b22fc01e3e6a95446895ddbd058a0ddfe43f5817c"}
Nov 02 22:52:52.604658 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:52.604632    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tr9pt" event={"ID":"fd2f118b-b69b-4b5e-a540-c28a023a7ab8","Type":"ContainerStarted","Data":"0b6814368f4216659a611800e1ee5e7220db78b5b8e63aa4279a696dc03d1ee3"}
Nov 02 22:52:52.606068 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:52.606048    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ngvq8" event={"ID":"19ca5d3a-9921-4eec-bb4b-cb8166fb85d5","Type":"ContainerStarted","Data":"ba1847ffb65654b9af674bd0718bb3973d08ebbe8a2ed4ebed28fb91e5cbd4b1"}
Nov 02 22:52:52.606156 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:52.606088    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ngvq8" event={"ID":"19ca5d3a-9921-4eec-bb4b-cb8166fb85d5","Type":"ContainerStarted","Data":"e17e12ef3e94df994d3ead5ce7d5e5e00a0ccf69244068ba70abbcb8bcecf529"}
Nov 02 22:52:53.068617 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:53.068577    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:53.222451 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:53.222410    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:53.612145 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:53.612105    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ngvq8" event={"ID":"19ca5d3a-9921-4eec-bb4b-cb8166fb85d5","Type":"ContainerStarted","Data":"c62a64354d2b1971219e7ecdd8acaee82b9ef3e7fc4c393c737ddfef0eb78523"}
Nov 02 22:52:53.768271 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:53.768203    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:53.930877 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:53.930840    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:54.122152 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.122061    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:54.125307 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.125283    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j"]
Nov 02 22:52:54.127357 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.127336    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j"
Nov 02 22:52:54.225852 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.225817    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 02 22:52:54.226040 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.225900    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wz9qf\""
Nov 02 22:52:54.226040 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.225913    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 22:52:54.226040 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.225953    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 02 22:52:54.226233 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.226105    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 22:52:54.226291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.226229    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 22:52:54.226669 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.226650    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5fab475f-2b50-4d87-8c50-298ec2a59016-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-c4z4j\" (UID: \"5fab475f-2b50-4d87-8c50-298ec2a59016\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j"
Nov 02 22:52:54.226750 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.226687    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fab475f-2b50-4d87-8c50-298ec2a59016-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-c4z4j\" (UID: \"5fab475f-2b50-4d87-8c50-298ec2a59016\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j"
Nov 02 22:52:54.226750 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.226715    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5fab475f-2b50-4d87-8c50-298ec2a59016-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-c4z4j\" (UID: \"5fab475f-2b50-4d87-8c50-298ec2a59016\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j"
Nov 02 22:52:54.226853 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.226768    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6gp7\" (UniqueName: \"kubernetes.io/projected/5fab475f-2b50-4d87-8c50-298ec2a59016-kube-api-access-w6gp7\") pod \"prometheus-operator-9cd89b74c-c4z4j\" (UID: \"5fab475f-2b50-4d87-8c50-298ec2a59016\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j"
Nov 02 22:52:54.275667 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.275638    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j"]
Nov 02 22:52:54.289218 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.288837    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:54.327909 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.327870    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fab475f-2b50-4d87-8c50-298ec2a59016-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-c4z4j\" (UID: \"5fab475f-2b50-4d87-8c50-298ec2a59016\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j"
Nov 02 22:52:54.328179 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.327933    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5fab475f-2b50-4d87-8c50-298ec2a59016-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-c4z4j\" (UID: \"5fab475f-2b50-4d87-8c50-298ec2a59016\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j"
Nov 02 22:52:54.328179 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.327971    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6gp7\" (UniqueName: \"kubernetes.io/projected/5fab475f-2b50-4d87-8c50-298ec2a59016-kube-api-access-w6gp7\") pod \"prometheus-operator-9cd89b74c-c4z4j\" (UID: \"5fab475f-2b50-4d87-8c50-298ec2a59016\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j"
Nov 02 22:52:54.328179 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.328045    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5fab475f-2b50-4d87-8c50-298ec2a59016-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-c4z4j\" (UID: \"5fab475f-2b50-4d87-8c50-298ec2a59016\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j"
Nov 02 22:52:54.329316 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.329289    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fab475f-2b50-4d87-8c50-298ec2a59016-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-c4z4j\" (UID: \"5fab475f-2b50-4d87-8c50-298ec2a59016\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j"
Nov 02 22:52:54.330684 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.330650    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5fab475f-2b50-4d87-8c50-298ec2a59016-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-c4z4j\" (UID: \"5fab475f-2b50-4d87-8c50-298ec2a59016\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j"
Nov 02 22:52:54.331327 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.331297    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5fab475f-2b50-4d87-8c50-298ec2a59016-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-c4z4j\" (UID: \"5fab475f-2b50-4d87-8c50-298ec2a59016\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j"
Nov 02 22:52:54.388374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.388343    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6gp7\" (UniqueName: \"kubernetes.io/projected/5fab475f-2b50-4d87-8c50-298ec2a59016-kube-api-access-w6gp7\") pod \"prometheus-operator-9cd89b74c-c4z4j\" (UID: \"5fab475f-2b50-4d87-8c50-298ec2a59016\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j"
Nov 02 22:52:54.440456 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.440434    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j"
Nov 02 22:52:54.618610 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.618571    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r4lgf" event={"ID":"de1dc693-c309-4d7c-b8dd-e1e7a435e8a1","Type":"ContainerStarted","Data":"a071d8608860693e0848cd07d628f25e6f78818cf8edfbe808c34f0a447f4a7e"}
Nov 02 22:52:54.619008 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.618638    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r4lgf" event={"ID":"de1dc693-c309-4d7c-b8dd-e1e7a435e8a1","Type":"ContainerStarted","Data":"ecf631fa0755de8ad570a6c0c87d75b76598ea7f5c943c9c1c9166f15526c38f"}
Nov 02 22:52:54.619008 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.618754    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r4lgf"
Nov 02 22:52:54.728385 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.728290    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-r4lgf" podStartSLOduration=2.272888427 podStartE2EDuration="4.728274833s" totalImagesPullingTime="2.455386406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:54.727791494 +0000 UTC m=+14.296146282" watchObservedRunningTime="2026-11-02 22:52:54.728274833 +0000 UTC m=+14.296629621"
Nov 02 22:52:54.876903 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:54.876634    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j"]
Nov 02 22:52:54.880183 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:52:54.880149    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fab475f_2b50_4d87_8c50_298ec2a59016.slice/crio-6e1e9a06814b9237c0450a3424b97215724cd874352a8eef2e68e77bd18455da WatchSource:0}: Error finding container 6e1e9a06814b9237c0450a3424b97215724cd874352a8eef2e68e77bd18455da: Status 404 returned error can't find the container with id 6e1e9a06814b9237c0450a3424b97215724cd874352a8eef2e68e77bd18455da
Nov 02 22:52:55.624471 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:55.624438    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ngvq8" event={"ID":"19ca5d3a-9921-4eec-bb4b-cb8166fb85d5","Type":"ContainerStarted","Data":"8879763381c24af12b122761a5eb9947ca5d841da5975226a10252e2c14f2965"}
Nov 02 22:52:55.625899 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:55.625876    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tr9pt" event={"ID":"fd2f118b-b69b-4b5e-a540-c28a023a7ab8","Type":"ContainerStarted","Data":"a9bdd10b9f93f00d20d355226d29d10edf8c8ef3b96aa34977fc4266f4433fa8"}
Nov 02 22:52:55.627017 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:55.626993    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j" event={"ID":"5fab475f-2b50-4d87-8c50-298ec2a59016","Type":"ContainerStarted","Data":"6e1e9a06814b9237c0450a3424b97215724cd874352a8eef2e68e77bd18455da"}
Nov 02 22:52:55.758854 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:55.758802    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ngvq8" podStartSLOduration=2.338392974 podStartE2EDuration="5.758786514s" totalImagesPullingTime="3.42039354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:52:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:55.758210803 +0000 UTC m=+15.326565590" watchObservedRunningTime="2026-11-02 22:52:55.758786514 +0000 UTC m=+15.327141356"
Nov 02 22:52:55.866698 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:55.866652    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tr9pt" podStartSLOduration=1.392252463 podStartE2EDuration="4.86663863s" totalImagesPullingTime="3.474386167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:55.866187038 +0000 UTC m=+15.434541825" watchObservedRunningTime="2026-11-02 22:52:55.86663863 +0000 UTC m=+15.434993416"
Nov 02 22:52:56.194079 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:56.193937    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:56.361996 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:56.361947    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:52:57.636128 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:57.636094    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j" event={"ID":"5fab475f-2b50-4d87-8c50-298ec2a59016","Type":"ContainerStarted","Data":"d1e28cf7da6cd39c2462c3ad7002d3d3b060dee613167a0e2d5735256185005c"}
Nov 02 22:52:57.636483 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:57.636156    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j" event={"ID":"5fab475f-2b50-4d87-8c50-298ec2a59016","Type":"ContainerStarted","Data":"1ee0cb07b48d7ea0617514f4db3911b35a6ab8c440137c1e833dca38ed9c48ba"}
Nov 02 22:52:57.745606 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:57.745554    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-c4z4j" podStartSLOduration=1.9265264439999998 podStartE2EDuration="3.745539529s" totalImagesPullingTime="1.819013085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:52:57.744576628 +0000 UTC m=+17.312931414" watchObservedRunningTime="2026-11-02 22:52:57.745539529 +0000 UTC m=+17.313894307"
Nov 02 22:52:59.288118 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:52:59.288072    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:00.073542 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.073492    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:00.234627 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.234594    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:00.401135 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.401046    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:00.560660 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.560623    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:00.732956 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.732867    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:00.783675 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.783648    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4smcv"]
Nov 02 22:53:00.785796 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.785780    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.873271 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.873243    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a8c5208-7b7f-4eb0-a083-bfad09651a37-node-exporter-wtmp\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.873271 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.873275    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a8c5208-7b7f-4eb0-a083-bfad09651a37-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.873484 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.873313    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvbn4\" (UniqueName: \"kubernetes.io/projected/0a8c5208-7b7f-4eb0-a083-bfad09651a37-kube-api-access-hvbn4\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.873484 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.873410    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a8c5208-7b7f-4eb0-a083-bfad09651a37-root\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.873549 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.873497    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a8c5208-7b7f-4eb0-a083-bfad09651a37-metrics-client-ca\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.873581 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.873557    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0a8c5208-7b7f-4eb0-a083-bfad09651a37-node-exporter-textfile\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.873616 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.873586    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a8c5208-7b7f-4eb0-a083-bfad09651a37-sys\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.873650 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.873628    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0a8c5208-7b7f-4eb0-a083-bfad09651a37-node-exporter-accelerators-collector-config\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.873712 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.873694    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a8c5208-7b7f-4eb0-a083-bfad09651a37-node-exporter-tls\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.901303 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.901271    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:00.940887 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.940855    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rnj8n\""
Nov 02 22:53:00.941056 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.940931    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 22:53:00.941056 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.940955    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 22:53:00.941123 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.941061    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 22:53:00.974991 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.974944    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a8c5208-7b7f-4eb0-a083-bfad09651a37-node-exporter-wtmp\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.975056 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.975014    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a8c5208-7b7f-4eb0-a083-bfad09651a37-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.975087 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.975059    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvbn4\" (UniqueName: \"kubernetes.io/projected/0a8c5208-7b7f-4eb0-a083-bfad09651a37-kube-api-access-hvbn4\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.975130 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.975084    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a8c5208-7b7f-4eb0-a083-bfad09651a37-root\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.975130 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.975112    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a8c5208-7b7f-4eb0-a083-bfad09651a37-metrics-client-ca\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.975235 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.975145    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0a8c5208-7b7f-4eb0-a083-bfad09651a37-node-exporter-wtmp\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.975235 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.975168    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0a8c5208-7b7f-4eb0-a083-bfad09651a37-node-exporter-textfile\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.975235 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.975212    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a8c5208-7b7f-4eb0-a083-bfad09651a37-sys\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.975375 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.975236    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0a8c5208-7b7f-4eb0-a083-bfad09651a37-node-exporter-accelerators-collector-config\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.975375 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.975274    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a8c5208-7b7f-4eb0-a083-bfad09651a37-node-exporter-tls\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.975375 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.975333    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a8c5208-7b7f-4eb0-a083-bfad09651a37-sys\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.975375 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.975306    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0a8c5208-7b7f-4eb0-a083-bfad09651a37-root\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.975803 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.975774    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a8c5208-7b7f-4eb0-a083-bfad09651a37-metrics-client-ca\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.975803 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.975789    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0a8c5208-7b7f-4eb0-a083-bfad09651a37-node-exporter-accelerators-collector-config\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.976017 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.975961    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0a8c5208-7b7f-4eb0-a083-bfad09651a37-node-exporter-textfile\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.977402 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.977384    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a8c5208-7b7f-4eb0-a083-bfad09651a37-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:00.977532 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:00.977515    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0a8c5208-7b7f-4eb0-a083-bfad09651a37-node-exporter-tls\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:01.028799 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:01.028775    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvbn4\" (UniqueName: \"kubernetes.io/projected/0a8c5208-7b7f-4eb0-a083-bfad09651a37-kube-api-access-hvbn4\") pod \"node-exporter-4smcv\" (UID: \"0a8c5208-7b7f-4eb0-a083-bfad09651a37\") " pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:01.097779 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:01.097748    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4smcv"
Nov 02 22:53:01.108553 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:01.108513    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:01.108895 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:53:01.108870    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8c5208_7b7f_4eb0_a083_bfad09651a37.slice/crio-caa6f9748670a41bf91802fb93d0a76b817a08b5d60889a22ac96b00fa935ea4 WatchSource:0}: Error finding container caa6f9748670a41bf91802fb93d0a76b817a08b5d60889a22ac96b00fa935ea4: Status 404 returned error can't find the container with id caa6f9748670a41bf91802fb93d0a76b817a08b5d60889a22ac96b00fa935ea4
Nov 02 22:53:01.382913 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:01.382826    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:01.651074 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:01.650995    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4smcv" event={"ID":"0a8c5208-7b7f-4eb0-a083-bfad09651a37","Type":"ContainerStarted","Data":"caa6f9748670a41bf91802fb93d0a76b817a08b5d60889a22ac96b00fa935ea4"}
Nov 02 22:53:01.997185 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:01.997155    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:02.179746 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.179712    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:02.808659 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.808603    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:02.857179 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.857156    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 22:53:02.859595 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.859580    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:02.909524 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.909497    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 22:53:02.909886 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.909853    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 22:53:02.910004 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.909917    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 22:53:02.910004 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.909958    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 22:53:02.955573 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.955548    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 22:53:02.955681 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.955590    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 22:53:02.955681 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.955657    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jrd64\""
Nov 02 22:53:02.955752 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.955684    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 22:53:02.990055 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.990006    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:02.990055 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.990046    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe891743-4f2d-4429-a053-5bc21bdf088f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:02.990149 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.990069    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:02.990149 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.990120    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe891743-4f2d-4429-a053-5bc21bdf088f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:02.990207 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.990174    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-web-config\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:02.990207 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.990197    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5x9f\" (UniqueName: \"kubernetes.io/projected/fe891743-4f2d-4429-a053-5bc21bdf088f-kube-api-access-w5x9f\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:02.990264 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.990215    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe891743-4f2d-4429-a053-5bc21bdf088f-config-out\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:02.990264 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.990233    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe891743-4f2d-4429-a053-5bc21bdf088f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:02.990333 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.990315    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:02.990393 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.990380    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-config-volume\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:02.990423 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.990410    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:02.990457 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.990430    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:02.990489 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:02.990471    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fe891743-4f2d-4429-a053-5bc21bdf088f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.006367 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.006347    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 22:53:03.009870 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.009854    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 22:53:03.009957 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.009946    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 22:53:03.091086 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.091055    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5x9f\" (UniqueName: \"kubernetes.io/projected/fe891743-4f2d-4429-a053-5bc21bdf088f-kube-api-access-w5x9f\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.091086 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.091090    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe891743-4f2d-4429-a053-5bc21bdf088f-config-out\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.091251 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.091107    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe891743-4f2d-4429-a053-5bc21bdf088f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.091251 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.091234    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.091377 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.091269    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-config-volume\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.091377 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.091303    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.091377 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.091338    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.091377 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.091363    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fe891743-4f2d-4429-a053-5bc21bdf088f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.091582 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.091401    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.091638 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.091603    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe891743-4f2d-4429-a053-5bc21bdf088f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.091688 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.091657    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.091751 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.091693    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe891743-4f2d-4429-a053-5bc21bdf088f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.091751 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.091723    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-web-config\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.091927 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.091865    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fe891743-4f2d-4429-a053-5bc21bdf088f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.092678 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.092307    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe891743-4f2d-4429-a053-5bc21bdf088f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.093056 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.093031    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe891743-4f2d-4429-a053-5bc21bdf088f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.094072 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.094045    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe891743-4f2d-4429-a053-5bc21bdf088f-config-out\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.094714 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.094357    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.094714 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.094421    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.094895 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.094724    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.094895 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.094847    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-web-config\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.095027 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.094995    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-config-volume\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.095086 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.095042    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.095762 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.095742    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe891743-4f2d-4429-a053-5bc21bdf088f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.095904 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.095887    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.145321 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.145300    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5x9f\" (UniqueName: \"kubernetes.io/projected/fe891743-4f2d-4429-a053-5bc21bdf088f-kube-api-access-w5x9f\") pod \"alertmanager-main-0\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.222290 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.222259    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:03.658995 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.658905    4294 generic.go:356] "Generic (PLEG): container finished" podID="0a8c5208-7b7f-4eb0-a083-bfad09651a37" containerID="4bf21c47a5c898192101f3d336e2feff9fa6cfe966bd41312a0fb72b805582f0" exitCode=0
Nov 02 22:53:03.659144 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.659004    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4smcv" event={"ID":"0a8c5208-7b7f-4eb0-a083-bfad09651a37","Type":"ContainerDied","Data":"4bf21c47a5c898192101f3d336e2feff9fa6cfe966bd41312a0fb72b805582f0"}
Nov 02 22:53:03.676617 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.676596    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 22:53:03.700790 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.700747    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:03.709776 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:53:03.709753    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe891743_4f2d_4429_a053_5bc21bdf088f.slice/crio-0399d8201732c398d1c5ea23312e26128a4745f87ae7fa808da10442fb6698b4 WatchSource:0}: Error finding container 0399d8201732c398d1c5ea23312e26128a4745f87ae7fa808da10442fb6698b4: Status 404 returned error can't find the container with id 0399d8201732c398d1c5ea23312e26128a4745f87ae7fa808da10442fb6698b4
Nov 02 22:53:03.861701 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:03.861664    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:04.025674 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:04.025635    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:04.633424 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:04.633389    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r4lgf"
Nov 02 22:53:04.663548 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:04.663517    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe891743-4f2d-4429-a053-5bc21bdf088f","Type":"ContainerStarted","Data":"0399d8201732c398d1c5ea23312e26128a4745f87ae7fa808da10442fb6698b4"}
Nov 02 22:53:04.665529 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:04.665506    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4smcv" event={"ID":"0a8c5208-7b7f-4eb0-a083-bfad09651a37","Type":"ContainerStarted","Data":"33d9e2b137411b764030db0f1e329ff1b3a8b0962c449058d33dd07a9c6a3972"}
Nov 02 22:53:04.665663 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:04.665549    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4smcv" event={"ID":"0a8c5208-7b7f-4eb0-a083-bfad09651a37","Type":"ContainerStarted","Data":"d2245e6870f0a9351a40e446eabf57184250952557fc93c9d8388ab5195d34c1"}
Nov 02 22:53:04.704187 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:04.704153    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:04.854147 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:04.854085    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4smcv" podStartSLOduration=3.171679631 podStartE2EDuration="4.854072195s" totalImagesPullingTime="1.614620965s" totalInitContainerRuntime="67.771599ms" isStatefulPod=true podCreationTimestamp="2026-11-02 22:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:04.853193431 +0000 UTC m=+24.421548217" watchObservedRunningTime="2026-11-02 22:53:04.854072195 +0000 UTC m=+24.422426982"
Nov 02 22:53:04.905382 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:04.905300    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:05.057328 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.057290    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:05.214975 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.214895    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:05.373856 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.373833    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:05.538999 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.538948    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:05.606097 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.606064    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-74ccc8f57c-bfffb"]
Nov 02 22:53:05.607951 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.607936    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.661443 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.661413    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 02 22:53:05.662134 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.662114    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-74ccc8f57c-bfffb"]
Nov 02 22:53:05.670261 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.670234    4294 generic.go:356] "Generic (PLEG): container finished" podID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerID="a2d538ca4c6c3cf76a044fe8011dba0a8b22ba4bd0d8c14eba7b22ad382dac48" exitCode=0
Nov 02 22:53:05.670375 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.670323    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe891743-4f2d-4429-a053-5bc21bdf088f","Type":"ContainerDied","Data":"a2d538ca4c6c3cf76a044fe8011dba0a8b22ba4bd0d8c14eba7b22ad382dac48"}
Nov 02 22:53:05.699930 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.699895    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:05.708532 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.708467    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 22:53:05.708663 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.708646    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 02 22:53:05.708721 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.708658    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-lsb2b\""
Nov 02 22:53:05.708721 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.708654    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 02 22:53:05.708721 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.708654    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.708721 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.708698    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5rrbnpi7b4rh9\""
Nov 02 22:53:05.708851 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.708722    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-metrics-server-audit-profiles\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.708851 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.708751    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-secret-metrics-server-client-certs\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.708851 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.708802    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-secret-metrics-server-tls\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.708851 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.708824    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4nrw\" (UniqueName: \"kubernetes.io/projected/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-kube-api-access-c4nrw\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.709016 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.708856    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-audit-log\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.709016 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.708895    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-client-ca-bundle\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.809521 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.809441    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-audit-log\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.809521 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.809474    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-client-ca-bundle\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.809721 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.809564    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.809721 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.809633    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-metrics-server-audit-profiles\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.809827 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.809726    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-secret-metrics-server-client-certs\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.809827 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.809802    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-secret-metrics-server-tls\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.809921 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.809860    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4nrw\" (UniqueName: \"kubernetes.io/projected/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-kube-api-access-c4nrw\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.809921 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.809893    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-audit-log\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.810678 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.810387    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.810822 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.810803    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-metrics-server-audit-profiles\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.812234 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.812211    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-client-ca-bundle\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.812480 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.812462    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-secret-metrics-server-client-certs\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.812549 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.812506    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-secret-metrics-server-tls\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.863717 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.863693    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4nrw\" (UniqueName: \"kubernetes.io/projected/4f4a11bd-282b-42c6-bb44-943adbbcb4ab-kube-api-access-c4nrw\") pod \"metrics-server-74ccc8f57c-bfffb\" (UID: \"4f4a11bd-282b-42c6-bb44-943adbbcb4ab\") " pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:05.917707 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:05.917680    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:06.317964 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:06.317942    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-74ccc8f57c-bfffb"]
Nov 02 22:53:06.319655 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:53:06.319631    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4a11bd_282b_42c6_bb44_943adbbcb4ab.slice/crio-fdf5dca91c5a45b17bf38d4457337ddcece254d03351384799910bf424c299cb WatchSource:0}: Error finding container fdf5dca91c5a45b17bf38d4457337ddcece254d03351384799910bf424c299cb: Status 404 returned error can't find the container with id fdf5dca91c5a45b17bf38d4457337ddcece254d03351384799910bf424c299cb
Nov 02 22:53:06.674877 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:06.674803    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb" event={"ID":"4f4a11bd-282b-42c6-bb44-943adbbcb4ab","Type":"ContainerStarted","Data":"fdf5dca91c5a45b17bf38d4457337ddcece254d03351384799910bf424c299cb"}
Nov 02 22:53:07.623851 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.623818    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bb6c5bc6-d8rcp"]
Nov 02 22:53:07.625886 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.625862    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.676128 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.676102    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 22:53:07.676319 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.676293    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 02 22:53:07.724449 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.723755    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 02 22:53:07.724449 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.723779    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 02 22:53:07.724449 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.723818    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 02 22:53:07.724449 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.724090    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9lw2l\""
Nov 02 22:53:07.724449 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.724136    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 02 22:53:07.724725 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.724461    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-service-ca\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.724725 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.724492    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-config\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.724725 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.724521    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-trusted-ca-bundle\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.724725 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.724661    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twbsq\" (UniqueName: \"kubernetes.io/projected/50baecc3-1870-4458-ac4e-56fbdace0d4d-kube-api-access-twbsq\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.724725 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.724713    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-serving-cert\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.725043 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.724831    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50baecc3-1870-4458-ac4e-56fbdace0d4d-tmp\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.725043 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.724898    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-oauth-config\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.725043 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.724928    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-oauth-serving-cert\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.820947 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.820921    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 02 22:53:07.821364 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.821338    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 22:53:07.823216 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.823157    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bb6c5bc6-d8rcp"]
Nov 02 22:53:07.825429 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.825408    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-oauth-config\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.825527 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.825438    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-oauth-serving-cert\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.825527 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.825464    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-service-ca\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.825527 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.825479    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-config\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.825527 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.825497    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-trusted-ca-bundle\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.825527 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.825519    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twbsq\" (UniqueName: \"kubernetes.io/projected/50baecc3-1870-4458-ac4e-56fbdace0d4d-kube-api-access-twbsq\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.825761 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.825543    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-serving-cert\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.825761 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.825567    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50baecc3-1870-4458-ac4e-56fbdace0d4d-tmp\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.825930 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.825910    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50baecc3-1870-4458-ac4e-56fbdace0d4d-tmp\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.826604 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.826580    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-config\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.826847 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.826798    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-trusted-ca-bundle\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.826847 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.826811    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-service-ca\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.827275 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.827253    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-oauth-serving-cert\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.828684 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.828662    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-serving-cert\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.828884 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.828867    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-oauth-config\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.880169 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.880144    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twbsq\" (UniqueName: \"kubernetes.io/projected/50baecc3-1870-4458-ac4e-56fbdace0d4d-kube-api-access-twbsq\") pod \"console-7bb6c5bc6-d8rcp\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") " pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:07.938554 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:07.938532    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:08.070671 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:08.070642    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:08.227144 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:08.227060    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:08.303208 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:08.303178    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bb6c5bc6-d8rcp"]
Nov 02 22:53:08.336134 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:53:08.336102    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50baecc3_1870_4458_ac4e_56fbdace0d4d.slice/crio-830669b5ca9da860f70a4aefc5d4272668f789db0c35f1292792f486de62c116 WatchSource:0}: Error finding container 830669b5ca9da860f70a4aefc5d4272668f789db0c35f1292792f486de62c116: Status 404 returned error can't find the container with id 830669b5ca9da860f70a4aefc5d4272668f789db0c35f1292792f486de62c116
Nov 02 22:53:08.686060 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:08.686017    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe891743-4f2d-4429-a053-5bc21bdf088f","Type":"ContainerStarted","Data":"2f6429048253c265b30e4a26d35b058021aad9545dd9b4b722b90088b1d76201"}
Nov 02 22:53:08.686483 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:08.686097    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe891743-4f2d-4429-a053-5bc21bdf088f","Type":"ContainerStarted","Data":"313f88707abb00d50450a0a78ccbfc719a36014bef3d758fd3d82161ce0ab656"}
Nov 02 22:53:08.686483 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:08.686138    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe891743-4f2d-4429-a053-5bc21bdf088f","Type":"ContainerStarted","Data":"f2cd5c0d2324e37aec6bc4c54f71378c05d118bad990ce64c3b7d02cfbda93fe"}
Nov 02 22:53:08.686483 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:08.686162    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe891743-4f2d-4429-a053-5bc21bdf088f","Type":"ContainerStarted","Data":"d61bc4c465252068cc36e3028d63ce51c9eba1a9a9468261c14085efe88e99e6"}
Nov 02 22:53:08.686483 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:08.686194    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe891743-4f2d-4429-a053-5bc21bdf088f","Type":"ContainerStarted","Data":"0ac2ee4f9d334ce612c1af0059bc0d561fd5a0ad07fb0e6de6ed00202c7c74c1"}
Nov 02 22:53:08.687259 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:08.687236    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb6c5bc6-d8rcp" event={"ID":"50baecc3-1870-4458-ac4e-56fbdace0d4d","Type":"ContainerStarted","Data":"830669b5ca9da860f70a4aefc5d4272668f789db0c35f1292792f486de62c116"}
Nov 02 22:53:08.688767 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:08.688744    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb" event={"ID":"4f4a11bd-282b-42c6-bb44-943adbbcb4ab","Type":"ContainerStarted","Data":"89e027697dd7b3a465e29be5837fb163abc4db638288e2fa358a98b0c2c95040"}
Nov 02 22:53:08.801620 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:08.801566    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb" podStartSLOduration=1.752947225 podStartE2EDuration="3.801552399s" totalImagesPullingTime="2.048605174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:08.801041598 +0000 UTC m=+28.369396384" watchObservedRunningTime="2026-11-02 22:53:08.801552399 +0000 UTC m=+28.369907186"
Nov 02 22:53:09.442321 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:09.442287    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qhbsn"
Nov 02 22:53:09.546030 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:09.545943    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:10.294936 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:10.294862    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:10.699845 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:10.699763    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe891743-4f2d-4429-a053-5bc21bdf088f","Type":"ContainerStarted","Data":"b5931233cc0d3e66a81ab862bb7c06ca1f7ace4280592c624c22246f5acd70b0"}
Nov 02 22:53:11.080195 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:11.080138    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-295c4"
Nov 02 22:53:11.214887 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:11.214810    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.567818461 podStartE2EDuration="9.214792327s" totalImagesPullingTime="5.630795542s" totalInitContainerRuntime="16.178324ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:11.003298496 +0000 UTC m=+30.571653283" watchObservedRunningTime="2026-11-02 22:53:11.214792327 +0000 UTC m=+30.783147117"
Nov 02 22:53:11.664165 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:11.664121    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:11.748830 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:53:11.748793    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 22:53:11.849800 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:11.849767    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:12.016831 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:12.016803    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:12.708091 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:12.708016    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb6c5bc6-d8rcp" event={"ID":"50baecc3-1870-4458-ac4e-56fbdace0d4d","Type":"ContainerStarted","Data":"bdfa9ca99b594a004f5365774cf21061250fc036dcdd424d546ad385a664cf72"}
Nov 02 22:53:12.823470 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:12.823416    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7bb6c5bc6-d8rcp" podStartSLOduration=1.947494785 podStartE2EDuration="5.823402796s" totalImagesPullingTime="3.875908011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:12.822530055 +0000 UTC m=+32.390884842" watchObservedRunningTime="2026-11-02 22:53:12.823402796 +0000 UTC m=+32.391757582"
Nov 02 22:53:13.904193 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:13.904157    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:14.081291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:14.081245    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:17.550658 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:17.550622    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:17.718310 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:17.718269    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:17.939719 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:17.939640    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:17.939719 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:17.939705    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:17.944486 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:17.944464    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:18.513707 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:18.513646    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:18.713632 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:18.713597    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:18.732104 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:18.732072    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:53:20.208407 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:20.208378    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:20.409659 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:20.409629    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:21.176853 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:21.176817    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:21.377606 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:21.377577    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:21.547557 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:21.547527    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:21.715066 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:21.715032    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:22.370359 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:22.370316    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:23.069500 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:23.069463    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:23.296394 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:23.296365    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:25.918402 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:25.918310    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:25.918402 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:25.918384    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:41.748421 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:53:41.748381    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 22:53:45.923306 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:45.923272    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:45.927201 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:45.927172    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-74ccc8f57c-bfffb"
Nov 02 22:53:52.152499 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:52.152465    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:52.344446 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:52.344409    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:53.077000 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:53.076950    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:53.240328 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:53.240290    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:53:54.247489 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:54.247451    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 22:53:54.247965 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:54.247786    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="alertmanager" containerID="cri-o://0ac2ee4f9d334ce612c1af0059bc0d561fd5a0ad07fb0e6de6ed00202c7c74c1" gracePeriod=120
Nov 02 22:53:54.247965 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:54.247837    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="kube-rbac-proxy-metric" containerID="cri-o://2f6429048253c265b30e4a26d35b058021aad9545dd9b4b722b90088b1d76201" gracePeriod=120
Nov 02 22:53:54.247965 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:54.247893    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="prom-label-proxy" containerID="cri-o://b5931233cc0d3e66a81ab862bb7c06ca1f7ace4280592c624c22246f5acd70b0" gracePeriod=120
Nov 02 22:53:54.247965 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:54.247886    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="kube-rbac-proxy-web" containerID="cri-o://f2cd5c0d2324e37aec6bc4c54f71378c05d118bad990ce64c3b7d02cfbda93fe" gracePeriod=120
Nov 02 22:53:54.247965 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:54.247921    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="kube-rbac-proxy" containerID="cri-o://313f88707abb00d50450a0a78ccbfc719a36014bef3d758fd3d82161ce0ab656" gracePeriod=120
Nov 02 22:53:54.247965 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:54.247948    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="config-reloader" containerID="cri-o://d61bc4c465252068cc36e3028d63ce51c9eba1a9a9468261c14085efe88e99e6" gracePeriod=120
Nov 02 22:53:54.844951 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:54.844919    4294 generic.go:356] "Generic (PLEG): container finished" podID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerID="b5931233cc0d3e66a81ab862bb7c06ca1f7ace4280592c624c22246f5acd70b0" exitCode=0
Nov 02 22:53:54.844951 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:54.844946    4294 generic.go:356] "Generic (PLEG): container finished" podID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerID="2f6429048253c265b30e4a26d35b058021aad9545dd9b4b722b90088b1d76201" exitCode=0
Nov 02 22:53:54.844951 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:54.844952    4294 generic.go:356] "Generic (PLEG): container finished" podID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerID="313f88707abb00d50450a0a78ccbfc719a36014bef3d758fd3d82161ce0ab656" exitCode=0
Nov 02 22:53:54.844951 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:54.844957    4294 generic.go:356] "Generic (PLEG): container finished" podID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerID="d61bc4c465252068cc36e3028d63ce51c9eba1a9a9468261c14085efe88e99e6" exitCode=0
Nov 02 22:53:54.844951 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:54.844962    4294 generic.go:356] "Generic (PLEG): container finished" podID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerID="0ac2ee4f9d334ce612c1af0059bc0d561fd5a0ad07fb0e6de6ed00202c7c74c1" exitCode=0
Nov 02 22:53:54.845255 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:54.844998    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe891743-4f2d-4429-a053-5bc21bdf088f","Type":"ContainerDied","Data":"b5931233cc0d3e66a81ab862bb7c06ca1f7ace4280592c624c22246f5acd70b0"}
Nov 02 22:53:54.845255 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:54.845062    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe891743-4f2d-4429-a053-5bc21bdf088f","Type":"ContainerDied","Data":"2f6429048253c265b30e4a26d35b058021aad9545dd9b4b722b90088b1d76201"}
Nov 02 22:53:54.845255 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:54.845088    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe891743-4f2d-4429-a053-5bc21bdf088f","Type":"ContainerDied","Data":"313f88707abb00d50450a0a78ccbfc719a36014bef3d758fd3d82161ce0ab656"}
Nov 02 22:53:54.845255 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:54.845111    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe891743-4f2d-4429-a053-5bc21bdf088f","Type":"ContainerDied","Data":"d61bc4c465252068cc36e3028d63ce51c9eba1a9a9468261c14085efe88e99e6"}
Nov 02 22:53:54.845255 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:54.845134    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe891743-4f2d-4429-a053-5bc21bdf088f","Type":"ContainerDied","Data":"0ac2ee4f9d334ce612c1af0059bc0d561fd5a0ad07fb0e6de6ed00202c7c74c1"}
Nov 02 22:53:55.487919 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.487895    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:55.586972 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.586890    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy\") pod \"fe891743-4f2d-4429-a053-5bc21bdf088f\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") "
Nov 02 22:53:55.586972 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.586943    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy-web\") pod \"fe891743-4f2d-4429-a053-5bc21bdf088f\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") "
Nov 02 22:53:55.587180 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.587001    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe891743-4f2d-4429-a053-5bc21bdf088f-kube-api-access-w5x9f\" (UniqueName: \"kubernetes.io/projected/fe891743-4f2d-4429-a053-5bc21bdf088f-kube-api-access-w5x9f\") pod \"fe891743-4f2d-4429-a053-5bc21bdf088f\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") "
Nov 02 22:53:55.587180 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.587021    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe891743-4f2d-4429-a053-5bc21bdf088f-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe891743-4f2d-4429-a053-5bc21bdf088f-alertmanager-trusted-ca-bundle\") pod \"fe891743-4f2d-4429-a053-5bc21bdf088f\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") "
Nov 02 22:53:55.587180 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.587050    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe891743-4f2d-4429-a053-5bc21bdf088f-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fe891743-4f2d-4429-a053-5bc21bdf088f-alertmanager-main-db\") pod \"fe891743-4f2d-4429-a053-5bc21bdf088f\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") "
Nov 02 22:53:55.587180 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.587087    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"fe891743-4f2d-4429-a053-5bc21bdf088f\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") "
Nov 02 22:53:55.587180 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.587120    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-main-tls\") pod \"fe891743-4f2d-4429-a053-5bc21bdf088f\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") "
Nov 02 22:53:55.587180 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.587149    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe891743-4f2d-4429-a053-5bc21bdf088f-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe891743-4f2d-4429-a053-5bc21bdf088f-metrics-client-ca\") pod \"fe891743-4f2d-4429-a053-5bc21bdf088f\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") "
Nov 02 22:53:55.587180 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.587176    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-config-volume\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-config-volume\") pod \"fe891743-4f2d-4429-a053-5bc21bdf088f\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") "
Nov 02 22:53:55.587579 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.587210    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe891743-4f2d-4429-a053-5bc21bdf088f-config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe891743-4f2d-4429-a053-5bc21bdf088f-config-out\") pod \"fe891743-4f2d-4429-a053-5bc21bdf088f\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") "
Nov 02 22:53:55.587579 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.587240    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-cluster-tls-config\") pod \"fe891743-4f2d-4429-a053-5bc21bdf088f\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") "
Nov 02 22:53:55.587579 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.587277    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe891743-4f2d-4429-a053-5bc21bdf088f-tls-assets\" (UniqueName: \"kubernetes.io/projected/fe891743-4f2d-4429-a053-5bc21bdf088f-tls-assets\") pod \"fe891743-4f2d-4429-a053-5bc21bdf088f\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") "
Nov 02 22:53:55.587579 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.587315    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-web-config\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-web-config\") pod \"fe891743-4f2d-4429-a053-5bc21bdf088f\" (UID: \"fe891743-4f2d-4429-a053-5bc21bdf088f\") "
Nov 02 22:53:55.587579 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.587459    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe891743-4f2d-4429-a053-5bc21bdf088f-alertmanager-trusted-ca-bundle" pod "fe891743-4f2d-4429-a053-5bc21bdf088f" (UID: "fe891743-4f2d-4429-a053-5bc21bdf088f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:53:55.587841 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.587580    4294 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe891743-4f2d-4429-a053-5bc21bdf088f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:53:55.587841 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.587610    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe891743-4f2d-4429-a053-5bc21bdf088f-metrics-client-ca" pod "fe891743-4f2d-4429-a053-5bc21bdf088f" (UID: "fe891743-4f2d-4429-a053-5bc21bdf088f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:53:55.587841 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.587729    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe891743-4f2d-4429-a053-5bc21bdf088f-alertmanager-main-db" pod "fe891743-4f2d-4429-a053-5bc21bdf088f" (UID: "fe891743-4f2d-4429-a053-5bc21bdf088f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:53:55.590212 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.590179    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy" pod "fe891743-4f2d-4429-a053-5bc21bdf088f" (UID: "fe891743-4f2d-4429-a053-5bc21bdf088f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:53:55.590528 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.590481    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy-web" pod "fe891743-4f2d-4429-a053-5bc21bdf088f" (UID: "fe891743-4f2d-4429-a053-5bc21bdf088f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:53:55.590809 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.590785    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe891743-4f2d-4429-a053-5bc21bdf088f-tls-assets" pod "fe891743-4f2d-4429-a053-5bc21bdf088f" (UID: "fe891743-4f2d-4429-a053-5bc21bdf088f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:53:55.590809 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.590793    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-main-tls" pod "fe891743-4f2d-4429-a053-5bc21bdf088f" (UID: "fe891743-4f2d-4429-a053-5bc21bdf088f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:53:55.590967 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.590919    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy-metric" pod "fe891743-4f2d-4429-a053-5bc21bdf088f" (UID: "fe891743-4f2d-4429-a053-5bc21bdf088f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:53:55.591132 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.591105    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe891743-4f2d-4429-a053-5bc21bdf088f-config-out" pod "fe891743-4f2d-4429-a053-5bc21bdf088f" (UID: "fe891743-4f2d-4429-a053-5bc21bdf088f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:53:55.591212 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.591124    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe891743-4f2d-4429-a053-5bc21bdf088f-kube-api-access-w5x9f" pod "fe891743-4f2d-4429-a053-5bc21bdf088f" (UID: "fe891743-4f2d-4429-a053-5bc21bdf088f"). InnerVolumeSpecName "kube-api-access-w5x9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:53:55.591212 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.591154    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-config-volume" pod "fe891743-4f2d-4429-a053-5bc21bdf088f" (UID: "fe891743-4f2d-4429-a053-5bc21bdf088f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:53:55.594355 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.594307    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-cluster-tls-config" pod "fe891743-4f2d-4429-a053-5bc21bdf088f" (UID: "fe891743-4f2d-4429-a053-5bc21bdf088f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:53:55.601401 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.601379    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-web-config" pod "fe891743-4f2d-4429-a053-5bc21bdf088f" (UID: "fe891743-4f2d-4429-a053-5bc21bdf088f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:53:55.688530 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.688490    4294 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-main-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:53:55.688530 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.688520    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe891743-4f2d-4429-a053-5bc21bdf088f-metrics-client-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:53:55.688530 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.688532    4294 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:53:55.688530 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.688541    4294 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe891743-4f2d-4429-a053-5bc21bdf088f-config-out\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:53:55.688794 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.688550    4294 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-cluster-tls-config\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:53:55.688794 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.688558    4294 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe891743-4f2d-4429-a053-5bc21bdf088f-tls-assets\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:53:55.688794 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.688568    4294 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-web-config\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:53:55.688794 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.688579    4294 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:53:55.688794 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.688589    4294 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:53:55.688794 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.688597    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5x9f\" (UniqueName: \"kubernetes.io/projected/fe891743-4f2d-4429-a053-5bc21bdf088f-kube-api-access-w5x9f\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:53:55.688794 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.688605    4294 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fe891743-4f2d-4429-a053-5bc21bdf088f-alertmanager-main-db\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:53:55.688794 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.688614    4294 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fe891743-4f2d-4429-a053-5bc21bdf088f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:53:55.851202 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.851125    4294 generic.go:356] "Generic (PLEG): container finished" podID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerID="f2cd5c0d2324e37aec6bc4c54f71378c05d118bad990ce64c3b7d02cfbda93fe" exitCode=0
Nov 02 22:53:55.851202 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.851177    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe891743-4f2d-4429-a053-5bc21bdf088f","Type":"ContainerDied","Data":"f2cd5c0d2324e37aec6bc4c54f71378c05d118bad990ce64c3b7d02cfbda93fe"}
Nov 02 22:53:55.851376 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.851221    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:55.851376 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.851234    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe891743-4f2d-4429-a053-5bc21bdf088f","Type":"ContainerDied","Data":"0399d8201732c398d1c5ea23312e26128a4745f87ae7fa808da10442fb6698b4"}
Nov 02 22:53:55.851376 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.851267    4294 scope.go:122] "RemoveContainer" containerID="b5931233cc0d3e66a81ab862bb7c06ca1f7ace4280592c624c22246f5acd70b0"
Nov 02 22:53:55.859541 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.859493    4294 scope.go:122] "RemoveContainer" containerID="2f6429048253c265b30e4a26d35b058021aad9545dd9b4b722b90088b1d76201"
Nov 02 22:53:55.866577 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.866554    4294 scope.go:122] "RemoveContainer" containerID="313f88707abb00d50450a0a78ccbfc719a36014bef3d758fd3d82161ce0ab656"
Nov 02 22:53:55.873026 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.873008    4294 scope.go:122] "RemoveContainer" containerID="f2cd5c0d2324e37aec6bc4c54f71378c05d118bad990ce64c3b7d02cfbda93fe"
Nov 02 22:53:55.879007 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.878973    4294 scope.go:122] "RemoveContainer" containerID="d61bc4c465252068cc36e3028d63ce51c9eba1a9a9468261c14085efe88e99e6"
Nov 02 22:53:55.885350 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.885334    4294 scope.go:122] "RemoveContainer" containerID="0ac2ee4f9d334ce612c1af0059bc0d561fd5a0ad07fb0e6de6ed00202c7c74c1"
Nov 02 22:53:55.891490 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.891474    4294 scope.go:122] "RemoveContainer" containerID="a2d538ca4c6c3cf76a044fe8011dba0a8b22ba4bd0d8c14eba7b22ad382dac48"
Nov 02 22:53:55.897650 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.897636    4294 scope.go:122] "RemoveContainer" containerID="b5931233cc0d3e66a81ab862bb7c06ca1f7ace4280592c624c22246f5acd70b0"
Nov 02 22:53:55.897921 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:53:55.897901    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5931233cc0d3e66a81ab862bb7c06ca1f7ace4280592c624c22246f5acd70b0\": container with ID starting with b5931233cc0d3e66a81ab862bb7c06ca1f7ace4280592c624c22246f5acd70b0 not found: ID does not exist" containerID="b5931233cc0d3e66a81ab862bb7c06ca1f7ace4280592c624c22246f5acd70b0"
Nov 02 22:53:55.897996 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.897930    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5931233cc0d3e66a81ab862bb7c06ca1f7ace4280592c624c22246f5acd70b0"} err="failed to get container status \"b5931233cc0d3e66a81ab862bb7c06ca1f7ace4280592c624c22246f5acd70b0\": rpc error: code = NotFound desc = could not find container \"b5931233cc0d3e66a81ab862bb7c06ca1f7ace4280592c624c22246f5acd70b0\": container with ID starting with b5931233cc0d3e66a81ab862bb7c06ca1f7ace4280592c624c22246f5acd70b0 not found: ID does not exist"
Nov 02 22:53:55.897996 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.897996    4294 scope.go:122] "RemoveContainer" containerID="2f6429048253c265b30e4a26d35b058021aad9545dd9b4b722b90088b1d76201"
Nov 02 22:53:55.898307 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:53:55.898285    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f6429048253c265b30e4a26d35b058021aad9545dd9b4b722b90088b1d76201\": container with ID starting with 2f6429048253c265b30e4a26d35b058021aad9545dd9b4b722b90088b1d76201 not found: ID does not exist" containerID="2f6429048253c265b30e4a26d35b058021aad9545dd9b4b722b90088b1d76201"
Nov 02 22:53:55.898384 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.898316    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6429048253c265b30e4a26d35b058021aad9545dd9b4b722b90088b1d76201"} err="failed to get container status \"2f6429048253c265b30e4a26d35b058021aad9545dd9b4b722b90088b1d76201\": rpc error: code = NotFound desc = could not find container \"2f6429048253c265b30e4a26d35b058021aad9545dd9b4b722b90088b1d76201\": container with ID starting with 2f6429048253c265b30e4a26d35b058021aad9545dd9b4b722b90088b1d76201 not found: ID does not exist"
Nov 02 22:53:55.898384 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.898339    4294 scope.go:122] "RemoveContainer" containerID="313f88707abb00d50450a0a78ccbfc719a36014bef3d758fd3d82161ce0ab656"
Nov 02 22:53:55.898607 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:53:55.898589    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313f88707abb00d50450a0a78ccbfc719a36014bef3d758fd3d82161ce0ab656\": container with ID starting with 313f88707abb00d50450a0a78ccbfc719a36014bef3d758fd3d82161ce0ab656 not found: ID does not exist" containerID="313f88707abb00d50450a0a78ccbfc719a36014bef3d758fd3d82161ce0ab656"
Nov 02 22:53:55.898644 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.898613    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313f88707abb00d50450a0a78ccbfc719a36014bef3d758fd3d82161ce0ab656"} err="failed to get container status \"313f88707abb00d50450a0a78ccbfc719a36014bef3d758fd3d82161ce0ab656\": rpc error: code = NotFound desc = could not find container \"313f88707abb00d50450a0a78ccbfc719a36014bef3d758fd3d82161ce0ab656\": container with ID starting with 313f88707abb00d50450a0a78ccbfc719a36014bef3d758fd3d82161ce0ab656 not found: ID does not exist"
Nov 02 22:53:55.898644 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.898627    4294 scope.go:122] "RemoveContainer" containerID="f2cd5c0d2324e37aec6bc4c54f71378c05d118bad990ce64c3b7d02cfbda93fe"
Nov 02 22:53:55.898848 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:53:55.898833    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2cd5c0d2324e37aec6bc4c54f71378c05d118bad990ce64c3b7d02cfbda93fe\": container with ID starting with f2cd5c0d2324e37aec6bc4c54f71378c05d118bad990ce64c3b7d02cfbda93fe not found: ID does not exist" containerID="f2cd5c0d2324e37aec6bc4c54f71378c05d118bad990ce64c3b7d02cfbda93fe"
Nov 02 22:53:55.898890 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.898851    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2cd5c0d2324e37aec6bc4c54f71378c05d118bad990ce64c3b7d02cfbda93fe"} err="failed to get container status \"f2cd5c0d2324e37aec6bc4c54f71378c05d118bad990ce64c3b7d02cfbda93fe\": rpc error: code = NotFound desc = could not find container \"f2cd5c0d2324e37aec6bc4c54f71378c05d118bad990ce64c3b7d02cfbda93fe\": container with ID starting with f2cd5c0d2324e37aec6bc4c54f71378c05d118bad990ce64c3b7d02cfbda93fe not found: ID does not exist"
Nov 02 22:53:55.898890 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.898867    4294 scope.go:122] "RemoveContainer" containerID="d61bc4c465252068cc36e3028d63ce51c9eba1a9a9468261c14085efe88e99e6"
Nov 02 22:53:55.899089 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:53:55.899074    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61bc4c465252068cc36e3028d63ce51c9eba1a9a9468261c14085efe88e99e6\": container with ID starting with d61bc4c465252068cc36e3028d63ce51c9eba1a9a9468261c14085efe88e99e6 not found: ID does not exist" containerID="d61bc4c465252068cc36e3028d63ce51c9eba1a9a9468261c14085efe88e99e6"
Nov 02 22:53:55.899138 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.899093    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61bc4c465252068cc36e3028d63ce51c9eba1a9a9468261c14085efe88e99e6"} err="failed to get container status \"d61bc4c465252068cc36e3028d63ce51c9eba1a9a9468261c14085efe88e99e6\": rpc error: code = NotFound desc = could not find container \"d61bc4c465252068cc36e3028d63ce51c9eba1a9a9468261c14085efe88e99e6\": container with ID starting with d61bc4c465252068cc36e3028d63ce51c9eba1a9a9468261c14085efe88e99e6 not found: ID does not exist"
Nov 02 22:53:55.899138 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.899104    4294 scope.go:122] "RemoveContainer" containerID="0ac2ee4f9d334ce612c1af0059bc0d561fd5a0ad07fb0e6de6ed00202c7c74c1"
Nov 02 22:53:55.899285 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:53:55.899268    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac2ee4f9d334ce612c1af0059bc0d561fd5a0ad07fb0e6de6ed00202c7c74c1\": container with ID starting with 0ac2ee4f9d334ce612c1af0059bc0d561fd5a0ad07fb0e6de6ed00202c7c74c1 not found: ID does not exist" containerID="0ac2ee4f9d334ce612c1af0059bc0d561fd5a0ad07fb0e6de6ed00202c7c74c1"
Nov 02 22:53:55.899347 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.899292    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac2ee4f9d334ce612c1af0059bc0d561fd5a0ad07fb0e6de6ed00202c7c74c1"} err="failed to get container status \"0ac2ee4f9d334ce612c1af0059bc0d561fd5a0ad07fb0e6de6ed00202c7c74c1\": rpc error: code = NotFound desc = could not find container \"0ac2ee4f9d334ce612c1af0059bc0d561fd5a0ad07fb0e6de6ed00202c7c74c1\": container with ID starting with 0ac2ee4f9d334ce612c1af0059bc0d561fd5a0ad07fb0e6de6ed00202c7c74c1 not found: ID does not exist"
Nov 02 22:53:55.899347 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.899313    4294 scope.go:122] "RemoveContainer" containerID="a2d538ca4c6c3cf76a044fe8011dba0a8b22ba4bd0d8c14eba7b22ad382dac48"
Nov 02 22:53:55.899515 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:53:55.899495    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d538ca4c6c3cf76a044fe8011dba0a8b22ba4bd0d8c14eba7b22ad382dac48\": container with ID starting with a2d538ca4c6c3cf76a044fe8011dba0a8b22ba4bd0d8c14eba7b22ad382dac48 not found: ID does not exist" containerID="a2d538ca4c6c3cf76a044fe8011dba0a8b22ba4bd0d8c14eba7b22ad382dac48"
Nov 02 22:53:55.899550 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:55.899520    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d538ca4c6c3cf76a044fe8011dba0a8b22ba4bd0d8c14eba7b22ad382dac48"} err="failed to get container status \"a2d538ca4c6c3cf76a044fe8011dba0a8b22ba4bd0d8c14eba7b22ad382dac48\": rpc error: code = NotFound desc = could not find container \"a2d538ca4c6c3cf76a044fe8011dba0a8b22ba4bd0d8c14eba7b22ad382dac48\": container with ID starting with a2d538ca4c6c3cf76a044fe8011dba0a8b22ba4bd0d8c14eba7b22ad382dac48 not found: ID does not exist"
Nov 02 22:53:56.010213 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.010176    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 22:53:56.013489 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.013466    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 22:53:56.057731 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.057704    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 22:53:56.057932 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.057921    4294 cpu_manager.go:414] "containerMap: removing container" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="kube-rbac-proxy-web"
Nov 02 22:53:56.058008 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.057933    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="kube-rbac-proxy-web"
Nov 02 22:53:56.058008 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.057946    4294 cpu_manager.go:414] "containerMap: removing container" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="prom-label-proxy"
Nov 02 22:53:56.058008 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.057951    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="prom-label-proxy"
Nov 02 22:53:56.058008 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.057961    4294 cpu_manager.go:414] "containerMap: removing container" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="init-config-reloader"
Nov 02 22:53:56.058008 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.057966    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="init-config-reloader"
Nov 02 22:53:56.058008 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.057973    4294 cpu_manager.go:414] "containerMap: removing container" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="kube-rbac-proxy-metric"
Nov 02 22:53:56.058008 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.057999    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="kube-rbac-proxy-metric"
Nov 02 22:53:56.058236 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.058013    4294 cpu_manager.go:414] "containerMap: removing container" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="alertmanager"
Nov 02 22:53:56.058236 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.058019    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="alertmanager"
Nov 02 22:53:56.058236 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.058031    4294 cpu_manager.go:414] "containerMap: removing container" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="config-reloader"
Nov 02 22:53:56.058236 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.058035    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="config-reloader"
Nov 02 22:53:56.058236 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.058043    4294 cpu_manager.go:414] "containerMap: removing container" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="kube-rbac-proxy"
Nov 02 22:53:56.058236 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.058047    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="kube-rbac-proxy"
Nov 02 22:53:56.058236 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.058106    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="config-reloader"
Nov 02 22:53:56.058236 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.058114    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="kube-rbac-proxy"
Nov 02 22:53:56.058236 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.058119    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="kube-rbac-proxy-web"
Nov 02 22:53:56.058236 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.058125    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="prom-label-proxy"
Nov 02 22:53:56.058236 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.058131    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="kube-rbac-proxy-metric"
Nov 02 22:53:56.058236 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.058136    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" containerName="alertmanager"
Nov 02 22:53:56.061371 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.061354    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.092973 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.092941    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.092973 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.092973    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-web-config\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.093159 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.093009    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fc66b40-91ea-4f9f-896e-202f918a6bb1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.093159 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.093097    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.093222 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.093185    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.093222 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.093213    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-config-volume\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.093285 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.093242    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1fc66b40-91ea-4f9f-896e-202f918a6bb1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.093319 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.093289    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1fc66b40-91ea-4f9f-896e-202f918a6bb1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.093351 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.093326    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65dpc\" (UniqueName: \"kubernetes.io/projected/1fc66b40-91ea-4f9f-896e-202f918a6bb1-kube-api-access-65dpc\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.093383 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.093352    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.093383 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.093369    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fc66b40-91ea-4f9f-896e-202f918a6bb1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.093440 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.093385    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1fc66b40-91ea-4f9f-896e-202f918a6bb1-config-out\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.093440 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.093408    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.111953 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.111892    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 22:53:56.111953 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.111893    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 22:53:56.111953 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.111893    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 22:53:56.114333 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.114314    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 22:53:56.115842 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.115825    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 22:53:56.163716 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.163685    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 22:53:56.194327 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.194296    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.194327 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.194330    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fc66b40-91ea-4f9f-896e-202f918a6bb1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.194536 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.194348    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1fc66b40-91ea-4f9f-896e-202f918a6bb1-config-out\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.194536 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.194478    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.194620 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.194532    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.194620 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.194566    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-web-config\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.194620 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.194592    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fc66b40-91ea-4f9f-896e-202f918a6bb1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.194762 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.194620    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.194762 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.194680    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.194762 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.194709    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-config-volume\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.194762 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.194743    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1fc66b40-91ea-4f9f-896e-202f918a6bb1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.194949 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.194768    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1fc66b40-91ea-4f9f-896e-202f918a6bb1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.194949 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.194793    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65dpc\" (UniqueName: \"kubernetes.io/projected/1fc66b40-91ea-4f9f-896e-202f918a6bb1-kube-api-access-65dpc\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.195522 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.195195    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fc66b40-91ea-4f9f-896e-202f918a6bb1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.195522 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.195438    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1fc66b40-91ea-4f9f-896e-202f918a6bb1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.195722 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.195634    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fc66b40-91ea-4f9f-896e-202f918a6bb1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.197110 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.197089    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-web-config\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.197190 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.197135    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.197450 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.197355    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1fc66b40-91ea-4f9f-896e-202f918a6bb1-config-out\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.197450 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.197440    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1fc66b40-91ea-4f9f-896e-202f918a6bb1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.197599 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.197401    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.213211 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.213184    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 22:53:56.213211 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.213206    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 22:53:56.213211 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.213185    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 22:53:56.213429 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.213249    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 22:53:56.213429 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.213332    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jrd64\""
Nov 02 22:53:56.217614 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.217586    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.217717 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.217623    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.217717 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.217674    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.217717 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.217699    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1fc66b40-91ea-4f9f-896e-202f918a6bb1-config-volume\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.249917 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.249886    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65dpc\" (UniqueName: \"kubernetes.io/projected/1fc66b40-91ea-4f9f-896e-202f918a6bb1-kube-api-access-65dpc\") pod \"alertmanager-main-0\" (UID: \"1fc66b40-91ea-4f9f-896e-202f918a6bb1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.376253 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.376158    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 22:53:56.545759 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.545724    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe891743-4f2d-4429-a053-5bc21bdf088f" path="/var/lib/kubelet/pods/fe891743-4f2d-4429-a053-5bc21bdf088f/volumes"
Nov 02 22:53:56.787929 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.787880    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 22:53:56.790499 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:53:56.790458    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc66b40_91ea_4f9f_896e_202f918a6bb1.slice/crio-a2ce128fdedf1fb22fd359b2391202717673d51063fc040256e3913bf71a9c12 WatchSource:0}: Error finding container a2ce128fdedf1fb22fd359b2391202717673d51063fc040256e3913bf71a9c12: Status 404 returned error can't find the container with id a2ce128fdedf1fb22fd359b2391202717673d51063fc040256e3913bf71a9c12
Nov 02 22:53:56.860181 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.860153    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fc66b40-91ea-4f9f-896e-202f918a6bb1","Type":"ContainerStarted","Data":"4919bded60394bfa5d36196d7992e70e3f61fb97fb204bea330ad0b352a0363e"}
Nov 02 22:53:56.860271 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:56.860226    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fc66b40-91ea-4f9f-896e-202f918a6bb1","Type":"ContainerStarted","Data":"a2ce128fdedf1fb22fd359b2391202717673d51063fc040256e3913bf71a9c12"}
Nov 02 22:53:57.870045 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:57.870010    4294 generic.go:356] "Generic (PLEG): container finished" podID="1fc66b40-91ea-4f9f-896e-202f918a6bb1" containerID="4919bded60394bfa5d36196d7992e70e3f61fb97fb204bea330ad0b352a0363e" exitCode=0
Nov 02 22:53:57.870469 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:57.870065    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fc66b40-91ea-4f9f-896e-202f918a6bb1","Type":"ContainerDied","Data":"4919bded60394bfa5d36196d7992e70e3f61fb97fb204bea330ad0b352a0363e"}
Nov 02 22:53:57.870469 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:57.870115    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fc66b40-91ea-4f9f-896e-202f918a6bb1","Type":"ContainerStarted","Data":"36f06ca0f52219530f23b0214cdfe8dbd7d2dd2bf658b0a26d3a7e313d71b21d"}
Nov 02 22:53:57.870469 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:57.870140    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fc66b40-91ea-4f9f-896e-202f918a6bb1","Type":"ContainerStarted","Data":"78142b5ba6e586829d46a3a5bf0822aa6ab6a2eb81d73126d2df1e834681a16f"}
Nov 02 22:53:57.870469 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:57.870164    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fc66b40-91ea-4f9f-896e-202f918a6bb1","Type":"ContainerStarted","Data":"5076d616c65527eedd6f1d70fd27a235b7e47489137a966598c8cae740c83185"}
Nov 02 22:53:57.870469 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:57.870186    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fc66b40-91ea-4f9f-896e-202f918a6bb1","Type":"ContainerStarted","Data":"11b0a0e98183f9c9ef29ab64816df890e5977e03ae9461574d8765af560e612b"}
Nov 02 22:53:57.870469 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:57.870209    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fc66b40-91ea-4f9f-896e-202f918a6bb1","Type":"ContainerStarted","Data":"705e412ac940673ff07a8870765ed6cbfd5f6fb908e826e08ee13ae6c0127d36"}
Nov 02 22:53:57.870469 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:57.870233    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1fc66b40-91ea-4f9f-896e-202f918a6bb1","Type":"ContainerStarted","Data":"2a11db78d809ced79a53efc5f39e96cb90dcff7e517bfdf57788cec2a2173074"}
Nov 02 22:53:57.992049 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:57.991974    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.978723199 podStartE2EDuration="2.991961376s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.238177ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:53:57.99092 +0000 UTC m=+77.559274786" watchObservedRunningTime="2026-11-02 22:53:57.991961376 +0000 UTC m=+77.560316164"
Nov 02 22:53:58.075143 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.075105    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7498985c47-849bp"]
Nov 02 22:53:58.077699 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.077676    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.116756 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.116724    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6csj\" (UniqueName: \"kubernetes.io/projected/91387c11-b3d4-46cd-b9a7-fc658f49571e-kube-api-access-m6csj\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.116897 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.116794    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/91387c11-b3d4-46cd-b9a7-fc658f49571e-telemeter-client-tls\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.116897 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.116840    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91387c11-b3d4-46cd-b9a7-fc658f49571e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.116974 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.116938    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/91387c11-b3d4-46cd-b9a7-fc658f49571e-secret-telemeter-client\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.117052 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.116971    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/91387c11-b3d4-46cd-b9a7-fc658f49571e-federate-client-tls\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.117103 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.117065    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91387c11-b3d4-46cd-b9a7-fc658f49571e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.117156 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.117097    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91387c11-b3d4-46cd-b9a7-fc658f49571e-metrics-client-ca\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.117156 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.117126    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91387c11-b3d4-46cd-b9a7-fc658f49571e-serving-certs-ca-bundle\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.128242 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.128193    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 02 22:53:58.128337 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.128323    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 02 22:53:58.128387 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.128357    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 02 22:53:58.128463 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.128442    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 02 22:53:58.128520 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.128465    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-gxzm6\""
Nov 02 22:53:58.128520 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.128474    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 02 22:53:58.175820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.175782    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 02 22:53:58.177014 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.176995    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7498985c47-849bp"]
Nov 02 22:53:58.218242 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.218210    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/91387c11-b3d4-46cd-b9a7-fc658f49571e-telemeter-client-tls\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.218242 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.218245    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91387c11-b3d4-46cd-b9a7-fc658f49571e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.218464 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.218279    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/91387c11-b3d4-46cd-b9a7-fc658f49571e-secret-telemeter-client\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.218464 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.218303    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/91387c11-b3d4-46cd-b9a7-fc658f49571e-federate-client-tls\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.218464 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.218334    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91387c11-b3d4-46cd-b9a7-fc658f49571e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.218464 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.218357    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91387c11-b3d4-46cd-b9a7-fc658f49571e-metrics-client-ca\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.218464 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.218380    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91387c11-b3d4-46cd-b9a7-fc658f49571e-serving-certs-ca-bundle\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.218464 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.218432    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6csj\" (UniqueName: \"kubernetes.io/projected/91387c11-b3d4-46cd-b9a7-fc658f49571e-kube-api-access-m6csj\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.219425 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.219379    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91387c11-b3d4-46cd-b9a7-fc658f49571e-serving-certs-ca-bundle\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.219425 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.219399    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/91387c11-b3d4-46cd-b9a7-fc658f49571e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.219425 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.219396    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91387c11-b3d4-46cd-b9a7-fc658f49571e-metrics-client-ca\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.220962 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.220937    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91387c11-b3d4-46cd-b9a7-fc658f49571e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.221082 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.221070    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/91387c11-b3d4-46cd-b9a7-fc658f49571e-federate-client-tls\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.221230 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.221211    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/91387c11-b3d4-46cd-b9a7-fc658f49571e-telemeter-client-tls\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.221359 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.221340    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/91387c11-b3d4-46cd-b9a7-fc658f49571e-secret-telemeter-client\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.272867 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.272836    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6csj\" (UniqueName: \"kubernetes.io/projected/91387c11-b3d4-46cd-b9a7-fc658f49571e-kube-api-access-m6csj\") pod \"telemeter-client-7498985c47-849bp\" (UID: \"91387c11-b3d4-46cd-b9a7-fc658f49571e\") " pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.390161 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.390071    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7498985c47-849bp"
Nov 02 22:53:58.843851 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.843807    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7498985c47-849bp"]
Nov 02 22:53:58.845946 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:53:58.845913    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91387c11_b3d4_46cd_b9a7_fc658f49571e.slice/crio-b1ed2d4b88d33b8dc9775390118902c2772e7e253a44b5c2ca4eb02e60d7809f WatchSource:0}: Error finding container b1ed2d4b88d33b8dc9775390118902c2772e7e253a44b5c2ca4eb02e60d7809f: Status 404 returned error can't find the container with id b1ed2d4b88d33b8dc9775390118902c2772e7e253a44b5c2ca4eb02e60d7809f
Nov 02 22:53:58.874313 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:53:58.874282    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7498985c47-849bp" event={"ID":"91387c11-b3d4-46cd-b9a7-fc658f49571e","Type":"ContainerStarted","Data":"b1ed2d4b88d33b8dc9775390118902c2772e7e253a44b5c2ca4eb02e60d7809f"}
Nov 02 22:54:01.887910 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:01.887869    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7498985c47-849bp" event={"ID":"91387c11-b3d4-46cd-b9a7-fc658f49571e","Type":"ContainerStarted","Data":"a345dd30b818b03fb6c5fee6a6246e4f137dbfeb4cb1c7272d774fc8615915de"}
Nov 02 22:54:01.888319 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:01.887954    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7498985c47-849bp" event={"ID":"91387c11-b3d4-46cd-b9a7-fc658f49571e","Type":"ContainerStarted","Data":"50e9949d3bfebb5bb3186c3159a81cc7048eba361b71cfe5681c846c84594915"}
Nov 02 22:54:01.888319 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:01.887974    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7498985c47-849bp" event={"ID":"91387c11-b3d4-46cd-b9a7-fc658f49571e","Type":"ContainerStarted","Data":"cfcc8673a170bcc0ace804c43fc82953ae5c8521e121c92dea43d02c68bdeaa6"}
Nov 02 22:54:02.096362 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.096305    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7498985c47-849bp" podStartSLOduration=1.933536203 podStartE2EDuration="4.096291474s" totalImagesPullingTime="2.162755271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:53:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:02.04848608 +0000 UTC m=+81.616840878" watchObservedRunningTime="2026-11-02 22:54:02.096291474 +0000 UTC m=+81.664646261"
Nov 02 22:54:02.451361 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.451327    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-58d6d96f5d-dh42j"]
Nov 02 22:54:02.453963 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.453934    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.508364 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.508330    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58d6d96f5d-dh42j"]
Nov 02 22:54:02.553740 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.553711    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52566b2e-96a7-4033-b8bd-006981ef64b1-console-serving-cert\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.553740 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.553744    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-console-config\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.553948 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.553802    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-service-ca\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.553948 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.553877    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wqns\" (UniqueName: \"kubernetes.io/projected/52566b2e-96a7-4033-b8bd-006981ef64b1-kube-api-access-6wqns\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.553948 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.553929    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52566b2e-96a7-4033-b8bd-006981ef64b1-console-oauth-config\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.554063 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.554018    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-trusted-ca-bundle\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.554063 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.554050    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-oauth-serving-cert\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.554129 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.554121    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52566b2e-96a7-4033-b8bd-006981ef64b1-tmp\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.654993 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.654951    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-trusted-ca-bundle\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.655146 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.655013    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-oauth-serving-cert\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.655223 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.655197    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52566b2e-96a7-4033-b8bd-006981ef64b1-tmp\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.655270 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.655256    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52566b2e-96a7-4033-b8bd-006981ef64b1-console-serving-cert\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.655305 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.655288    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-console-config\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.655341 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.655321    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-service-ca\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.655388 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.655358    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wqns\" (UniqueName: \"kubernetes.io/projected/52566b2e-96a7-4033-b8bd-006981ef64b1-kube-api-access-6wqns\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.655469 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.655452    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52566b2e-96a7-4033-b8bd-006981ef64b1-console-oauth-config\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.655592 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.655571    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52566b2e-96a7-4033-b8bd-006981ef64b1-tmp\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.655770 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.655747    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-oauth-serving-cert\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.655942 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.655918    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-trusted-ca-bundle\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.656019 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.655968    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-service-ca\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.656271 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.656253    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-console-config\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.657801 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.657781    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52566b2e-96a7-4033-b8bd-006981ef64b1-console-oauth-config\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.657921 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.657901    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52566b2e-96a7-4033-b8bd-006981ef64b1-console-serving-cert\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.709848 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.709756    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wqns\" (UniqueName: \"kubernetes.io/projected/52566b2e-96a7-4033-b8bd-006981ef64b1-kube-api-access-6wqns\") pod \"console-58d6d96f5d-dh42j\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") " pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.764703 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.764663    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:02.770363 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:02.770338    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:54:03.106209 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:03.106171    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:54:03.128665 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:03.128643    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58d6d96f5d-dh42j"]
Nov 02 22:54:03.130352 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:54:03.130328    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52566b2e_96a7_4033_b8bd_006981ef64b1.slice/crio-852b14f91ebb93bcca3d6ff237d98eac3143c3c2ae05b7a3bbfa020ab1b3943b WatchSource:0}: Error finding container 852b14f91ebb93bcca3d6ff237d98eac3143c3c2ae05b7a3bbfa020ab1b3943b: Status 404 returned error can't find the container with id 852b14f91ebb93bcca3d6ff237d98eac3143c3c2ae05b7a3bbfa020ab1b3943b
Nov 02 22:54:03.264726 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:03.264688    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:54:03.895552 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:03.895513    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d6d96f5d-dh42j" event={"ID":"52566b2e-96a7-4033-b8bd-006981ef64b1","Type":"ContainerStarted","Data":"bd97d2c3b7a93573a379255e4bb37fa5ee7006b6b5362e9fb11fe19b6f545e18"}
Nov 02 22:54:03.895733 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:03.895606    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d6d96f5d-dh42j" event={"ID":"52566b2e-96a7-4033-b8bd-006981ef64b1","Type":"ContainerStarted","Data":"852b14f91ebb93bcca3d6ff237d98eac3143c3c2ae05b7a3bbfa020ab1b3943b"}
Nov 02 22:54:04.006470 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:04.006409    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-58d6d96f5d-dh42j" podStartSLOduration=2.006396495 podStartE2EDuration="2.006396495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:04.005450214 +0000 UTC m=+83.573805003" watchObservedRunningTime="2026-11-02 22:54:04.006396495 +0000 UTC m=+83.574751281"
Nov 02 22:54:05.891628 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:05.891593    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:54:06.055469 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:06.055438    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:54:11.747118 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:54:11.747081    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 22:54:12.765143 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:12.765106    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:12.765522 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:12.765168    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:12.769779 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:12.769755    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:12.931641 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:12.931611    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:54:13.090524 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:13.090445    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bb6c5bc6-d8rcp"]
Nov 02 22:54:19.838528 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:19.838491    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:54:20.000678 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:20.000645    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:54:21.433486 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:21.433447    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:54:32.483694 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:32.483658    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:54:32.642033 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:32.641975    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:54:38.064640 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.064607    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:54:38.110158 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.110121    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7bb6c5bc6-d8rcp" podUID="50baecc3-1870-4458-ac4e-56fbdace0d4d" containerName="console" containerID="cri-o://bdfa9ca99b594a004f5365774cf21061250fc036dcdd424d546ad385a664cf72" gracePeriod=15
Nov 02 22:54:38.231129 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.231103    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:54:38.347907 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.347882    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:54:38.450737 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.450703    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50baecc3-1870-4458-ac4e-56fbdace0d4d-kube-api-access-twbsq\" (UniqueName: \"kubernetes.io/projected/50baecc3-1870-4458-ac4e-56fbdace0d4d-kube-api-access-twbsq\") pod \"50baecc3-1870-4458-ac4e-56fbdace0d4d\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") "
Nov 02 22:54:38.450897 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.450744    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-config\" (UniqueName: \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-config\") pod \"50baecc3-1870-4458-ac4e-56fbdace0d4d\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") "
Nov 02 22:54:38.450897 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.450764    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-oauth-config\") pod \"50baecc3-1870-4458-ac4e-56fbdace0d4d\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") "
Nov 02 22:54:38.450897 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.450789    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-oauth-serving-cert\") pod \"50baecc3-1870-4458-ac4e-56fbdace0d4d\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") "
Nov 02 22:54:38.450897 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.450848    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50baecc3-1870-4458-ac4e-56fbdace0d4d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/50baecc3-1870-4458-ac4e-56fbdace0d4d-tmp\") pod \"50baecc3-1870-4458-ac4e-56fbdace0d4d\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") "
Nov 02 22:54:38.450897 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.450883    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-trusted-ca-bundle\") pod \"50baecc3-1870-4458-ac4e-56fbdace0d4d\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") "
Nov 02 22:54:38.451134 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.450940    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-serving-cert\") pod \"50baecc3-1870-4458-ac4e-56fbdace0d4d\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") "
Nov 02 22:54:38.451219 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.451191    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50baecc3-1870-4458-ac4e-56fbdace0d4d-tmp" pod "50baecc3-1870-4458-ac4e-56fbdace0d4d" (UID: "50baecc3-1870-4458-ac4e-56fbdace0d4d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:54:38.451296 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.451233    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-service-ca\" (UniqueName: \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-service-ca\") pod \"50baecc3-1870-4458-ac4e-56fbdace0d4d\" (UID: \"50baecc3-1870-4458-ac4e-56fbdace0d4d\") "
Nov 02 22:54:38.451455 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.451418    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-oauth-serving-cert" pod "50baecc3-1870-4458-ac4e-56fbdace0d4d" (UID: "50baecc3-1870-4458-ac4e-56fbdace0d4d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:54:38.451533 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.451464    4294 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50baecc3-1870-4458-ac4e-56fbdace0d4d-tmp\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:54:38.451533 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.451494    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-config" pod "50baecc3-1870-4458-ac4e-56fbdace0d4d" (UID: "50baecc3-1870-4458-ac4e-56fbdace0d4d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:54:38.451640 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.451583    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-service-ca" pod "50baecc3-1870-4458-ac4e-56fbdace0d4d" (UID: "50baecc3-1870-4458-ac4e-56fbdace0d4d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:54:38.451640 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.451597    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-trusted-ca-bundle" pod "50baecc3-1870-4458-ac4e-56fbdace0d4d" (UID: "50baecc3-1870-4458-ac4e-56fbdace0d4d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:54:38.453227 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.453207    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-serving-cert" pod "50baecc3-1870-4458-ac4e-56fbdace0d4d" (UID: "50baecc3-1870-4458-ac4e-56fbdace0d4d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:54:38.453406 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.453387    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-oauth-config" pod "50baecc3-1870-4458-ac4e-56fbdace0d4d" (UID: "50baecc3-1870-4458-ac4e-56fbdace0d4d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:54:38.453470 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.453388    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50baecc3-1870-4458-ac4e-56fbdace0d4d-kube-api-access-twbsq" pod "50baecc3-1870-4458-ac4e-56fbdace0d4d" (UID: "50baecc3-1870-4458-ac4e-56fbdace0d4d"). InnerVolumeSpecName "kube-api-access-twbsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:54:38.552135 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.552110    4294 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-serving-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:54:38.552135 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.552135    4294 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:54:38.552277 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.552145    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twbsq\" (UniqueName: \"kubernetes.io/projected/50baecc3-1870-4458-ac4e-56fbdace0d4d-kube-api-access-twbsq\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:54:38.552277 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.552156    4294 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-config\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:54:38.552277 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.552165    4294 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50baecc3-1870-4458-ac4e-56fbdace0d4d-console-oauth-config\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:54:38.552277 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.552173    4294 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-oauth-serving-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:54:38.552277 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.552181    4294 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50baecc3-1870-4458-ac4e-56fbdace0d4d-trusted-ca-bundle\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:54:38.811361 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.811330    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-566978dcc7-xcztq"]
Nov 02 22:54:38.811607 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.811595    4294 cpu_manager.go:414] "containerMap: removing container" podUID="50baecc3-1870-4458-ac4e-56fbdace0d4d" containerName="console"
Nov 02 22:54:38.811648 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.811608    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50baecc3-1870-4458-ac4e-56fbdace0d4d" containerName="console"
Nov 02 22:54:38.811682 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.811662    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="50baecc3-1870-4458-ac4e-56fbdace0d4d" containerName="console"
Nov 02 22:54:38.813405 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.813390    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.853543 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.853519    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-console-config\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.853686 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.853548    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-console-oauth-config\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.853686 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.853573    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-oauth-serving-cert\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.853686 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.853590    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft6vr\" (UniqueName: \"kubernetes.io/projected/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-kube-api-access-ft6vr\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.853784 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.853758    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-service-ca\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.853817 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.853794    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-console-serving-cert\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.853849 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.853817    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-tmp\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.853849 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.853842    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-trusted-ca-bundle\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.907099 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.907070    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-566978dcc7-xcztq"]
Nov 02 22:54:38.955170 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.955138    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-console-config\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.955170 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.955174    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-console-oauth-config\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.955366 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.955195    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-oauth-serving-cert\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.955366 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.955310    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft6vr\" (UniqueName: \"kubernetes.io/projected/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-kube-api-access-ft6vr\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.955446 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.955384    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-service-ca\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.955446 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.955427    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-console-serving-cert\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.955541 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.955450    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-tmp\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.955541 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.955496    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-trusted-ca-bundle\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.955996 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.955950    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-tmp\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.956128 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.956079    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-oauth-serving-cert\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.956225 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.956201    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-service-ca\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.956332 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.956313    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-trusted-ca-bundle\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.956370 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.956328    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-console-config\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.957659 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.957639    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-console-oauth-config\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:38.957844 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:38.957826    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-console-serving-cert\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:39.010075 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:39.010047    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft6vr\" (UniqueName: \"kubernetes.io/projected/c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0-kube-api-access-ft6vr\") pod \"console-566978dcc7-xcztq\" (UID: \"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0\") " pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:39.015740 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:39.015714    4294 generic.go:356] "Generic (PLEG): container finished" podID="50baecc3-1870-4458-ac4e-56fbdace0d4d" containerID="bdfa9ca99b594a004f5365774cf21061250fc036dcdd424d546ad385a664cf72" exitCode=0
Nov 02 22:54:39.015847 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:39.015775    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bb6c5bc6-d8rcp"
Nov 02 22:54:39.015847 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:39.015796    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb6c5bc6-d8rcp" event={"ID":"50baecc3-1870-4458-ac4e-56fbdace0d4d","Type":"ContainerDied","Data":"bdfa9ca99b594a004f5365774cf21061250fc036dcdd424d546ad385a664cf72"}
Nov 02 22:54:39.015919 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:39.015857    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bb6c5bc6-d8rcp" event={"ID":"50baecc3-1870-4458-ac4e-56fbdace0d4d","Type":"ContainerDied","Data":"830669b5ca9da860f70a4aefc5d4272668f789db0c35f1292792f486de62c116"}
Nov 02 22:54:39.015919 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:39.015879    4294 scope.go:122] "RemoveContainer" containerID="bdfa9ca99b594a004f5365774cf21061250fc036dcdd424d546ad385a664cf72"
Nov 02 22:54:39.023487 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:39.023469    4294 scope.go:122] "RemoveContainer" containerID="bdfa9ca99b594a004f5365774cf21061250fc036dcdd424d546ad385a664cf72"
Nov 02 22:54:39.023745 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:54:39.023724    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdfa9ca99b594a004f5365774cf21061250fc036dcdd424d546ad385a664cf72\": container with ID starting with bdfa9ca99b594a004f5365774cf21061250fc036dcdd424d546ad385a664cf72 not found: ID does not exist" containerID="bdfa9ca99b594a004f5365774cf21061250fc036dcdd424d546ad385a664cf72"
Nov 02 22:54:39.023817 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:39.023757    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdfa9ca99b594a004f5365774cf21061250fc036dcdd424d546ad385a664cf72"} err="failed to get container status \"bdfa9ca99b594a004f5365774cf21061250fc036dcdd424d546ad385a664cf72\": rpc error: code = NotFound desc = could not find container \"bdfa9ca99b594a004f5365774cf21061250fc036dcdd424d546ad385a664cf72\": container with ID starting with bdfa9ca99b594a004f5365774cf21061250fc036dcdd424d546ad385a664cf72 not found: ID does not exist"
Nov 02 22:54:39.102870 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:39.102770    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:54:39.123272 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:39.123237    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:39.172844 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:39.172790    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bb6c5bc6-d8rcp"]
Nov 02 22:54:39.176166 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:39.176136    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bb6c5bc6-d8rcp"]
Nov 02 22:54:39.260544 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:39.260513    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:54:39.453958 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:39.453876    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:54:39.482403 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:39.482381    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-566978dcc7-xcztq"]
Nov 02 22:54:39.484471 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:54:39.484444    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a3a4fd_99c9_438a_8a52_5b55af7fb3a0.slice/crio-ada03c0f5b0d792e9bcf4757279a49508af0f894b73520409bbe56e433aa4811 WatchSource:0}: Error finding container ada03c0f5b0d792e9bcf4757279a49508af0f894b73520409bbe56e433aa4811: Status 404 returned error can't find the container with id ada03c0f5b0d792e9bcf4757279a49508af0f894b73520409bbe56e433aa4811
Nov 02 22:54:39.618299 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:39.618259    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:54:40.021888 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:40.021850    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-566978dcc7-xcztq" event={"ID":"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0","Type":"ContainerStarted","Data":"9c103e429d5d5f29c3176201b974308fb3423f500aca2b566251723424f6cb98"}
Nov 02 22:54:40.021888 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:40.021896    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-566978dcc7-xcztq" event={"ID":"c1a3a4fd-99c9-438a-8a52-5b55af7fb3a0","Type":"ContainerStarted","Data":"ada03c0f5b0d792e9bcf4757279a49508af0f894b73520409bbe56e433aa4811"}
Nov 02 22:54:40.132119 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:40.132072    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-566978dcc7-xcztq" podStartSLOduration=2.132058919 podStartE2EDuration="2.132058919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:54:40.13158703 +0000 UTC m=+119.699941827" watchObservedRunningTime="2026-11-02 22:54:40.132058919 +0000 UTC m=+119.700413706"
Nov 02 22:54:40.544975 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:40.544944    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50baecc3-1870-4458-ac4e-56fbdace0d4d" path="/var/lib/kubelet/pods/50baecc3-1870-4458-ac4e-56fbdace0d4d/volumes"
Nov 02 22:54:41.747825 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:54:41.747785    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 22:54:49.123444 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:49.123406    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:49.123897 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:49.123464    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:49.128111 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:49.128089    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:50.058296 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:50.058267    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-566978dcc7-xcztq"
Nov 02 22:54:50.217597 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:50.217559    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58d6d96f5d-dh42j"]
Nov 02 22:54:51.803731 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:54:51.803694    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:55:11.747147 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:55:11.747071    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 22:55:15.236628 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.236571    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-58d6d96f5d-dh42j" podUID="52566b2e-96a7-4033-b8bd-006981ef64b1" containerName="console" containerID="cri-o://bd97d2c3b7a93573a379255e4bb37fa5ee7006b6b5362e9fb11fe19b6f545e18" gracePeriod=15
Nov 02 22:55:15.479579 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.479556    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:55:15.643907 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.643874    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52566b2e-96a7-4033-b8bd-006981ef64b1-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52566b2e-96a7-4033-b8bd-006981ef64b1-console-oauth-config\") pod \"52566b2e-96a7-4033-b8bd-006981ef64b1\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") "
Nov 02 22:55:15.644102 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.643914    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-service-ca\" (UniqueName: \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-service-ca\") pod \"52566b2e-96a7-4033-b8bd-006981ef64b1\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") "
Nov 02 22:55:15.644102 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.643937    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52566b2e-96a7-4033-b8bd-006981ef64b1-tmp\" (UniqueName: \"kubernetes.io/empty-dir/52566b2e-96a7-4033-b8bd-006981ef64b1-tmp\") pod \"52566b2e-96a7-4033-b8bd-006981ef64b1\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") "
Nov 02 22:55:15.644102 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.643992    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52566b2e-96a7-4033-b8bd-006981ef64b1-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52566b2e-96a7-4033-b8bd-006981ef64b1-console-serving-cert\") pod \"52566b2e-96a7-4033-b8bd-006981ef64b1\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") "
Nov 02 22:55:15.644102 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.644032    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-console-config\" (UniqueName: \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-console-config\") pod \"52566b2e-96a7-4033-b8bd-006981ef64b1\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") "
Nov 02 22:55:15.644102 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.644052    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-trusted-ca-bundle\") pod \"52566b2e-96a7-4033-b8bd-006981ef64b1\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") "
Nov 02 22:55:15.644349 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.644125    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-oauth-serving-cert\") pod \"52566b2e-96a7-4033-b8bd-006981ef64b1\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") "
Nov 02 22:55:15.644349 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.644168    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52566b2e-96a7-4033-b8bd-006981ef64b1-kube-api-access-6wqns\" (UniqueName: \"kubernetes.io/projected/52566b2e-96a7-4033-b8bd-006981ef64b1-kube-api-access-6wqns\") pod \"52566b2e-96a7-4033-b8bd-006981ef64b1\" (UID: \"52566b2e-96a7-4033-b8bd-006981ef64b1\") "
Nov 02 22:55:15.644349 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.644288    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52566b2e-96a7-4033-b8bd-006981ef64b1-tmp" pod "52566b2e-96a7-4033-b8bd-006981ef64b1" (UID: "52566b2e-96a7-4033-b8bd-006981ef64b1"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:55:15.644502 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.644368    4294 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52566b2e-96a7-4033-b8bd-006981ef64b1-tmp\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:55:15.644502 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.644452    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-service-ca" pod "52566b2e-96a7-4033-b8bd-006981ef64b1" (UID: "52566b2e-96a7-4033-b8bd-006981ef64b1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:55:15.644609 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.644539    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-console-config" pod "52566b2e-96a7-4033-b8bd-006981ef64b1" (UID: "52566b2e-96a7-4033-b8bd-006981ef64b1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:55:15.644664 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.644601    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-oauth-serving-cert" pod "52566b2e-96a7-4033-b8bd-006981ef64b1" (UID: "52566b2e-96a7-4033-b8bd-006981ef64b1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:55:15.644722 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.644695    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-trusted-ca-bundle" pod "52566b2e-96a7-4033-b8bd-006981ef64b1" (UID: "52566b2e-96a7-4033-b8bd-006981ef64b1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:55:15.646128 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.646105    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52566b2e-96a7-4033-b8bd-006981ef64b1-console-oauth-config" pod "52566b2e-96a7-4033-b8bd-006981ef64b1" (UID: "52566b2e-96a7-4033-b8bd-006981ef64b1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:55:15.646340 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.646315    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52566b2e-96a7-4033-b8bd-006981ef64b1-kube-api-access-6wqns" pod "52566b2e-96a7-4033-b8bd-006981ef64b1" (UID: "52566b2e-96a7-4033-b8bd-006981ef64b1"). InnerVolumeSpecName "kube-api-access-6wqns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:55:15.646441 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.646418    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52566b2e-96a7-4033-b8bd-006981ef64b1-console-serving-cert" pod "52566b2e-96a7-4033-b8bd-006981ef64b1" (UID: "52566b2e-96a7-4033-b8bd-006981ef64b1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:55:15.744910 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.744881    4294 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52566b2e-96a7-4033-b8bd-006981ef64b1-console-oauth-config\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:55:15.744910 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.744905    4294 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:55:15.744910 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.744914    4294 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52566b2e-96a7-4033-b8bd-006981ef64b1-console-serving-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:55:15.745138 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.744923    4294 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-console-config\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:55:15.745138 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.744932    4294 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-trusted-ca-bundle\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:55:15.745138 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.744940    4294 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52566b2e-96a7-4033-b8bd-006981ef64b1-oauth-serving-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:55:15.745138 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:15.744949    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wqns\" (UniqueName: \"kubernetes.io/projected/52566b2e-96a7-4033-b8bd-006981ef64b1-kube-api-access-6wqns\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:55:16.140083 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:16.140048    4294 generic.go:356] "Generic (PLEG): container finished" podID="52566b2e-96a7-4033-b8bd-006981ef64b1" containerID="bd97d2c3b7a93573a379255e4bb37fa5ee7006b6b5362e9fb11fe19b6f545e18" exitCode=0
Nov 02 22:55:16.140275 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:16.140116    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d6d96f5d-dh42j" event={"ID":"52566b2e-96a7-4033-b8bd-006981ef64b1","Type":"ContainerDied","Data":"bd97d2c3b7a93573a379255e4bb37fa5ee7006b6b5362e9fb11fe19b6f545e18"}
Nov 02 22:55:16.140275 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:16.140125    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d6d96f5d-dh42j"
Nov 02 22:55:16.140275 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:16.140155    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d6d96f5d-dh42j" event={"ID":"52566b2e-96a7-4033-b8bd-006981ef64b1","Type":"ContainerDied","Data":"852b14f91ebb93bcca3d6ff237d98eac3143c3c2ae05b7a3bbfa020ab1b3943b"}
Nov 02 22:55:16.140275 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:16.140175    4294 scope.go:122] "RemoveContainer" containerID="bd97d2c3b7a93573a379255e4bb37fa5ee7006b6b5362e9fb11fe19b6f545e18"
Nov 02 22:55:16.148693 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:16.148675    4294 scope.go:122] "RemoveContainer" containerID="bd97d2c3b7a93573a379255e4bb37fa5ee7006b6b5362e9fb11fe19b6f545e18"
Nov 02 22:55:16.148961 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:55:16.148944    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd97d2c3b7a93573a379255e4bb37fa5ee7006b6b5362e9fb11fe19b6f545e18\": container with ID starting with bd97d2c3b7a93573a379255e4bb37fa5ee7006b6b5362e9fb11fe19b6f545e18 not found: ID does not exist" containerID="bd97d2c3b7a93573a379255e4bb37fa5ee7006b6b5362e9fb11fe19b6f545e18"
Nov 02 22:55:16.149030 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:16.148969    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd97d2c3b7a93573a379255e4bb37fa5ee7006b6b5362e9fb11fe19b6f545e18"} err="failed to get container status \"bd97d2c3b7a93573a379255e4bb37fa5ee7006b6b5362e9fb11fe19b6f545e18\": rpc error: code = NotFound desc = could not find container \"bd97d2c3b7a93573a379255e4bb37fa5ee7006b6b5362e9fb11fe19b6f545e18\": container with ID starting with bd97d2c3b7a93573a379255e4bb37fa5ee7006b6b5362e9fb11fe19b6f545e18 not found: ID does not exist"
Nov 02 22:55:16.296467 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:16.296436    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58d6d96f5d-dh42j"]
Nov 02 22:55:16.344241 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:16.344214    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58d6d96f5d-dh42j"]
Nov 02 22:55:16.544944 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:16.544918    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52566b2e-96a7-4033-b8bd-006981ef64b1" path="/var/lib/kubelet/pods/52566b2e-96a7-4033-b8bd-006981ef64b1/volumes"
Nov 02 22:55:18.573757 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:18.573725    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:55:18.739391 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:18.739347    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:55:21.758664 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:21.758635    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:55:29.827059 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:55:29.827012    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:55:41.747210 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:55:41.747165    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 22:56:00.722930 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:00.722884    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:00.977422 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:00.977340    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:01.706332 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.706297    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp"]
Nov 02 22:56:01.706569 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.706557    4294 cpu_manager.go:414] "containerMap: removing container" podUID="52566b2e-96a7-4033-b8bd-006981ef64b1" containerName="console"
Nov 02 22:56:01.706654 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.706570    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52566b2e-96a7-4033-b8bd-006981ef64b1" containerName="console"
Nov 02 22:56:01.706654 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.706639    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="52566b2e-96a7-4033-b8bd-006981ef64b1" containerName="console"
Nov 02 22:56:01.709448 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.709429    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp"
Nov 02 22:56:01.776374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.776329    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84edfcc3-1664-4518-a95e-7a1b3c1817a4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp\" (UID: \"84edfcc3-1664-4518-a95e-7a1b3c1817a4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp"
Nov 02 22:56:01.776374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.776376    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95xkw\" (UniqueName: \"kubernetes.io/projected/84edfcc3-1664-4518-a95e-7a1b3c1817a4-kube-api-access-95xkw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp\" (UID: \"84edfcc3-1664-4518-a95e-7a1b3c1817a4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp"
Nov 02 22:56:01.776803 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.776482    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84edfcc3-1664-4518-a95e-7a1b3c1817a4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp\" (UID: \"84edfcc3-1664-4518-a95e-7a1b3c1817a4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp"
Nov 02 22:56:01.826113 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.826078    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 22:56:01.826113 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.826086    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-bw6kf\""
Nov 02 22:56:01.826113 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.826115    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 22:56:01.827462 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.827442    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp"]
Nov 02 22:56:01.879390 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.879354    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84edfcc3-1664-4518-a95e-7a1b3c1817a4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp\" (UID: \"84edfcc3-1664-4518-a95e-7a1b3c1817a4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp"
Nov 02 22:56:01.879591 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.879406    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95xkw\" (UniqueName: \"kubernetes.io/projected/84edfcc3-1664-4518-a95e-7a1b3c1817a4-kube-api-access-95xkw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp\" (UID: \"84edfcc3-1664-4518-a95e-7a1b3c1817a4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp"
Nov 02 22:56:01.879591 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.879471    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84edfcc3-1664-4518-a95e-7a1b3c1817a4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp\" (UID: \"84edfcc3-1664-4518-a95e-7a1b3c1817a4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp"
Nov 02 22:56:01.879815 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.879792    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84edfcc3-1664-4518-a95e-7a1b3c1817a4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp\" (UID: \"84edfcc3-1664-4518-a95e-7a1b3c1817a4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp"
Nov 02 22:56:01.879885 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.879837    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84edfcc3-1664-4518-a95e-7a1b3c1817a4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp\" (UID: \"84edfcc3-1664-4518-a95e-7a1b3c1817a4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp"
Nov 02 22:56:01.907010 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.906964    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt"]
Nov 02 22:56:01.910337 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.910320    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt"
Nov 02 22:56:01.934105 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.934074    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95xkw\" (UniqueName: \"kubernetes.io/projected/84edfcc3-1664-4518-a95e-7a1b3c1817a4-kube-api-access-95xkw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp\" (UID: \"84edfcc3-1664-4518-a95e-7a1b3c1817a4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp"
Nov 02 22:56:01.963845 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.963773    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt"]
Nov 02 22:56:01.980618 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.980580    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzh9g\" (UniqueName: \"kubernetes.io/projected/f3c88446-d04e-46cb-8f5b-61b76aac49b7-kube-api-access-bzh9g\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt\" (UID: \"f3c88446-d04e-46cb-8f5b-61b76aac49b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt"
Nov 02 22:56:01.980618 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.980622    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3c88446-d04e-46cb-8f5b-61b76aac49b7-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt\" (UID: \"f3c88446-d04e-46cb-8f5b-61b76aac49b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt"
Nov 02 22:56:01.980813 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.980737    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3c88446-d04e-46cb-8f5b-61b76aac49b7-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt\" (UID: \"f3c88446-d04e-46cb-8f5b-61b76aac49b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt"
Nov 02 22:56:01.996677 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:01.996636    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-szr7t\""
Nov 02 22:56:02.024752 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.024705    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp"
Nov 02 22:56:02.084312 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.084273    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzh9g\" (UniqueName: \"kubernetes.io/projected/f3c88446-d04e-46cb-8f5b-61b76aac49b7-kube-api-access-bzh9g\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt\" (UID: \"f3c88446-d04e-46cb-8f5b-61b76aac49b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt"
Nov 02 22:56:02.084943 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.084911    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3c88446-d04e-46cb-8f5b-61b76aac49b7-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt\" (UID: \"f3c88446-d04e-46cb-8f5b-61b76aac49b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt"
Nov 02 22:56:02.085419 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.085378    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3c88446-d04e-46cb-8f5b-61b76aac49b7-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt\" (UID: \"f3c88446-d04e-46cb-8f5b-61b76aac49b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt"
Nov 02 22:56:02.085642 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.085622    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3c88446-d04e-46cb-8f5b-61b76aac49b7-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt\" (UID: \"f3c88446-d04e-46cb-8f5b-61b76aac49b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt"
Nov 02 22:56:02.086103 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.085966    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3c88446-d04e-46cb-8f5b-61b76aac49b7-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt\" (UID: \"f3c88446-d04e-46cb-8f5b-61b76aac49b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt"
Nov 02 22:56:02.139060 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.139029    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzh9g\" (UniqueName: \"kubernetes.io/projected/f3c88446-d04e-46cb-8f5b-61b76aac49b7-kube-api-access-bzh9g\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt\" (UID: \"f3c88446-d04e-46cb-8f5b-61b76aac49b7\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt"
Nov 02 22:56:02.195456 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.195424    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"]
Nov 02 22:56:02.199454 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.199437    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.225170 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.225095    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt"
Nov 02 22:56:02.250141 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.250107    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 22:56:02.252700 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.252676    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 22:56:02.253092 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.253076    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tbzbd\""
Nov 02 22:56:02.253796 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.253783    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 22:56:02.254047 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.254035    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 22:56:02.254133 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.254121    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 22:56:02.258787 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.258758    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"]
Nov 02 22:56:02.287694 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.287663    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/35db14cd-79a6-46bd-93cb-2368586e9ccb-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.287838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.287711    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.287838 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.287802    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.287954 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.287861    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.288027 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.287962    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.288071 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.288040    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.288132 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.288115    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8zvp\" (UniqueName: \"kubernetes.io/projected/35db14cd-79a6-46bd-93cb-2368586e9ccb-kube-api-access-z8zvp\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.315346 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.315312    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:02.350041 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.350011    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 22:56:02.389393 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.389361    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.389540 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.389409    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.389540 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.389433    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.389540 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.389462    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.389540 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.389498    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8zvp\" (UniqueName: \"kubernetes.io/projected/35db14cd-79a6-46bd-93cb-2368586e9ccb-kube-api-access-z8zvp\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.389540 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.389531    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/35db14cd-79a6-46bd-93cb-2368586e9ccb-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.389785 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.389558    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.390140 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.390111    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.391738 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.391707    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/35db14cd-79a6-46bd-93cb-2368586e9ccb-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.391858 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.391844    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.392309 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.392286    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.392388 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.392286    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.433822 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.433799    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp"]
Nov 02 22:56:02.435413 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:56:02.435389    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84edfcc3_1664_4518_a95e_7a1b3c1817a4.slice/crio-4e94b14622d27ac86cdf7e15bbe85795777fcf152441a797bd968a3162978f63 WatchSource:0}: Error finding container 4e94b14622d27ac86cdf7e15bbe85795777fcf152441a797bd968a3162978f63: Status 404 returned error can't find the container with id 4e94b14622d27ac86cdf7e15bbe85795777fcf152441a797bd968a3162978f63
Nov 02 22:56:02.446159 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.446132    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.446159 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.446148    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8zvp\" (UniqueName: \"kubernetes.io/projected/35db14cd-79a6-46bd-93cb-2368586e9ccb-kube-api-access-z8zvp\") pod \"istiod-openshift-gateway-7876bfcb74-jfth9\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.511163 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.511127    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:02.527225 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.527194    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:02.588177 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.588152    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt"]
Nov 02 22:56:02.591124 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:56:02.591092    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c88446_d04e_46cb_8f5b_61b76aac49b7.slice/crio-dea9afb4f4a758ac98b312affead15bab0cdeba536f499efd6267832f6db9289 WatchSource:0}: Error finding container dea9afb4f4a758ac98b312affead15bab0cdeba536f499efd6267832f6db9289: Status 404 returned error can't find the container with id dea9afb4f4a758ac98b312affead15bab0cdeba536f499efd6267832f6db9289
Nov 02 22:56:02.877850 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:02.877821    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"]
Nov 02 22:56:02.878790 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:56:02.878760    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35db14cd_79a6_46bd_93cb_2368586e9ccb.slice/crio-b2bec09fcfdcaea9ee1016dc39ebaa88eee76b5fb451641f7c900976f8365e23 WatchSource:0}: Error finding container b2bec09fcfdcaea9ee1016dc39ebaa88eee76b5fb451641f7c900976f8365e23: Status 404 returned error can't find the container with id b2bec09fcfdcaea9ee1016dc39ebaa88eee76b5fb451641f7c900976f8365e23
Nov 02 22:56:03.161665 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:03.161548    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:03.293787 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:03.293750    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt" event={"ID":"f3c88446-d04e-46cb-8f5b-61b76aac49b7","Type":"ContainerStarted","Data":"dea9afb4f4a758ac98b312affead15bab0cdeba536f499efd6267832f6db9289"}
Nov 02 22:56:03.294920 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:03.294892    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9" event={"ID":"35db14cd-79a6-46bd-93cb-2368586e9ccb","Type":"ContainerStarted","Data":"b2bec09fcfdcaea9ee1016dc39ebaa88eee76b5fb451641f7c900976f8365e23"}
Nov 02 22:56:03.295936 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:03.295914    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp" event={"ID":"84edfcc3-1664-4518-a95e-7a1b3c1817a4","Type":"ContainerStarted","Data":"4e94b14622d27ac86cdf7e15bbe85795777fcf152441a797bd968a3162978f63"}
Nov 02 22:56:07.362672 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:07.362633    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 22:56:07.363074 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:07.362722    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 22:56:08.323544 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:08.323505    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9" event={"ID":"35db14cd-79a6-46bd-93cb-2368586e9ccb","Type":"ContainerStarted","Data":"7e774fcdf9463be473650b69fcac7ed9ae6d05eaf4695dd56607a3af87176be5"}
Nov 02 22:56:08.324045 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:08.324025    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:08.325778 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:08.325729    4294 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-jfth9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 22:56:08.325919 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:08.325797    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9" podUID="35db14cd-79a6-46bd-93cb-2368586e9ccb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 22:56:08.523022 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:08.522939    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9" podStartSLOduration=2.041954756 podStartE2EDuration="6.522921515s" totalImagesPullingTime="4.480966759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:08.483137104 +0000 UTC m=+208.051491887" watchObservedRunningTime="2026-11-02 22:56:08.522921515 +0000 UTC m=+208.091276303"
Nov 02 22:56:08.771152 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:08.771092    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:08.937717 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:08.937650    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:09.328883 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:09.328854    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:56:09.693387 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:09.693301    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:09.874967 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:09.874872    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:10.038859 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:10.038835    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:10.332337 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:10.332306    4294 generic.go:356] "Generic (PLEG): container finished" podID="f3c88446-d04e-46cb-8f5b-61b76aac49b7" containerID="a8e7858402129e69575b1b56fa01b3f3e42374b68b9b6c16664fea600097a6a5" exitCode=0
Nov 02 22:56:10.332487 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:10.332392    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt" event={"ID":"f3c88446-d04e-46cb-8f5b-61b76aac49b7","Type":"ContainerDied","Data":"a8e7858402129e69575b1b56fa01b3f3e42374b68b9b6c16664fea600097a6a5"}
Nov 02 22:56:10.334031 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:10.334010    4294 generic.go:356] "Generic (PLEG): container finished" podID="84edfcc3-1664-4518-a95e-7a1b3c1817a4" containerID="dadcaf005ef8502d3080fb406491e9a3ee75fcbb641b8e7fe5826c7952f1c2fc" exitCode=0
Nov 02 22:56:10.334131 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:10.334089    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp" event={"ID":"84edfcc3-1664-4518-a95e-7a1b3c1817a4","Type":"ContainerDied","Data":"dadcaf005ef8502d3080fb406491e9a3ee75fcbb641b8e7fe5826c7952f1c2fc"}
Nov 02 22:56:11.748058 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:56:11.748016    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 22:56:12.344303 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:12.344271    4294 generic.go:356] "Generic (PLEG): container finished" podID="f3c88446-d04e-46cb-8f5b-61b76aac49b7" containerID="c9b4ab10bc7feac918fafb0c0a7f05a88a1e0852ec623454a37016b5bca5669e" exitCode=0
Nov 02 22:56:12.344488 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:12.344320    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt" event={"ID":"f3c88446-d04e-46cb-8f5b-61b76aac49b7","Type":"ContainerDied","Data":"c9b4ab10bc7feac918fafb0c0a7f05a88a1e0852ec623454a37016b5bca5669e"}
Nov 02 22:56:13.525478 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:13.525433    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:14.354351 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:14.354252    4294 generic.go:356] "Generic (PLEG): container finished" podID="84edfcc3-1664-4518-a95e-7a1b3c1817a4" containerID="f8f705e0d3dcd86f0dd3008f4ffb57826c61d9e8b46b46028a7e9eed232227fc" exitCode=0
Nov 02 22:56:14.354351 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:14.354300    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp" event={"ID":"84edfcc3-1664-4518-a95e-7a1b3c1817a4","Type":"ContainerDied","Data":"f8f705e0d3dcd86f0dd3008f4ffb57826c61d9e8b46b46028a7e9eed232227fc"}
Nov 02 22:56:20.382913 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:20.382870    4294 generic.go:356] "Generic (PLEG): container finished" podID="f3c88446-d04e-46cb-8f5b-61b76aac49b7" containerID="9bfc20d00c76a1081ac581390173131c798d9047fd085925eaade12c250a780a" exitCode=0
Nov 02 22:56:20.383411 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:20.382959    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt" event={"ID":"f3c88446-d04e-46cb-8f5b-61b76aac49b7","Type":"ContainerDied","Data":"9bfc20d00c76a1081ac581390173131c798d9047fd085925eaade12c250a780a"}
Nov 02 22:56:21.508263 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:21.508235    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt"
Nov 02 22:56:21.684369 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:21.684279    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c88446-d04e-46cb-8f5b-61b76aac49b7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3c88446-d04e-46cb-8f5b-61b76aac49b7-bundle\") pod \"f3c88446-d04e-46cb-8f5b-61b76aac49b7\" (UID: \"f3c88446-d04e-46cb-8f5b-61b76aac49b7\") "
Nov 02 22:56:21.684369 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:21.684364    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3c88446-d04e-46cb-8f5b-61b76aac49b7-kube-api-access-bzh9g\" (UniqueName: \"kubernetes.io/projected/f3c88446-d04e-46cb-8f5b-61b76aac49b7-kube-api-access-bzh9g\") pod \"f3c88446-d04e-46cb-8f5b-61b76aac49b7\" (UID: \"f3c88446-d04e-46cb-8f5b-61b76aac49b7\") "
Nov 02 22:56:21.684586 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:21.684426    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c88446-d04e-46cb-8f5b-61b76aac49b7-util\" (UniqueName: \"kubernetes.io/empty-dir/f3c88446-d04e-46cb-8f5b-61b76aac49b7-util\") pod \"f3c88446-d04e-46cb-8f5b-61b76aac49b7\" (UID: \"f3c88446-d04e-46cb-8f5b-61b76aac49b7\") "
Nov 02 22:56:21.684915 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:21.684880    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c88446-d04e-46cb-8f5b-61b76aac49b7-bundle" pod "f3c88446-d04e-46cb-8f5b-61b76aac49b7" (UID: "f3c88446-d04e-46cb-8f5b-61b76aac49b7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:56:21.685098 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:21.685078    4294 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f3c88446-d04e-46cb-8f5b-61b76aac49b7-bundle\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:56:21.687313 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:21.687282    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c88446-d04e-46cb-8f5b-61b76aac49b7-kube-api-access-bzh9g" pod "f3c88446-d04e-46cb-8f5b-61b76aac49b7" (UID: "f3c88446-d04e-46cb-8f5b-61b76aac49b7"). InnerVolumeSpecName "kube-api-access-bzh9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:56:21.689936 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:21.689911    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c88446-d04e-46cb-8f5b-61b76aac49b7-util" pod "f3c88446-d04e-46cb-8f5b-61b76aac49b7" (UID: "f3c88446-d04e-46cb-8f5b-61b76aac49b7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:56:21.786449 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:21.786398    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzh9g\" (UniqueName: \"kubernetes.io/projected/f3c88446-d04e-46cb-8f5b-61b76aac49b7-kube-api-access-bzh9g\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:56:21.786449 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:21.786444    4294 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f3c88446-d04e-46cb-8f5b-61b76aac49b7-util\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:56:22.073260 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:22.073229    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:22.391678 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:22.391590    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt"
Nov 02 22:56:22.391678 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:22.391607    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71lnzgt" event={"ID":"f3c88446-d04e-46cb-8f5b-61b76aac49b7","Type":"ContainerDied","Data":"dea9afb4f4a758ac98b312affead15bab0cdeba536f499efd6267832f6db9289"}
Nov 02 22:56:22.391678 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:22.391661    4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dea9afb4f4a758ac98b312affead15bab0cdeba536f499efd6267832f6db9289"
Nov 02 22:56:22.393751 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:22.393723    4294 generic.go:356] "Generic (PLEG): container finished" podID="84edfcc3-1664-4518-a95e-7a1b3c1817a4" containerID="f0c595d8447f7f5c45cad44d5673aeaa00cfef75b696ab4d4106045442f6bd3b" exitCode=0
Nov 02 22:56:22.393876 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:22.393802    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp" event={"ID":"84edfcc3-1664-4518-a95e-7a1b3c1817a4","Type":"ContainerDied","Data":"f0c595d8447f7f5c45cad44d5673aeaa00cfef75b696ab4d4106045442f6bd3b"}
Nov 02 22:56:23.527879 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:23.527845    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp"
Nov 02 22:56:23.599600 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:23.599566    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84edfcc3-1664-4518-a95e-7a1b3c1817a4-kube-api-access-95xkw\" (UniqueName: \"kubernetes.io/projected/84edfcc3-1664-4518-a95e-7a1b3c1817a4-kube-api-access-95xkw\") pod \"84edfcc3-1664-4518-a95e-7a1b3c1817a4\" (UID: \"84edfcc3-1664-4518-a95e-7a1b3c1817a4\") "
Nov 02 22:56:23.599757 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:23.599636    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84edfcc3-1664-4518-a95e-7a1b3c1817a4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/84edfcc3-1664-4518-a95e-7a1b3c1817a4-bundle\") pod \"84edfcc3-1664-4518-a95e-7a1b3c1817a4\" (UID: \"84edfcc3-1664-4518-a95e-7a1b3c1817a4\") "
Nov 02 22:56:23.599757 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:23.599741    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84edfcc3-1664-4518-a95e-7a1b3c1817a4-util\" (UniqueName: \"kubernetes.io/empty-dir/84edfcc3-1664-4518-a95e-7a1b3c1817a4-util\") pod \"84edfcc3-1664-4518-a95e-7a1b3c1817a4\" (UID: \"84edfcc3-1664-4518-a95e-7a1b3c1817a4\") "
Nov 02 22:56:23.600051 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:23.600032    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84edfcc3-1664-4518-a95e-7a1b3c1817a4-bundle" pod "84edfcc3-1664-4518-a95e-7a1b3c1817a4" (UID: "84edfcc3-1664-4518-a95e-7a1b3c1817a4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:56:23.601710 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:23.601690    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84edfcc3-1664-4518-a95e-7a1b3c1817a4-kube-api-access-95xkw" pod "84edfcc3-1664-4518-a95e-7a1b3c1817a4" (UID: "84edfcc3-1664-4518-a95e-7a1b3c1817a4"). InnerVolumeSpecName "kube-api-access-95xkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:56:23.604736 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:23.604714    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84edfcc3-1664-4518-a95e-7a1b3c1817a4-util" pod "84edfcc3-1664-4518-a95e-7a1b3c1817a4" (UID: "84edfcc3-1664-4518-a95e-7a1b3c1817a4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:56:23.700800 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:23.700690    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95xkw\" (UniqueName: \"kubernetes.io/projected/84edfcc3-1664-4518-a95e-7a1b3c1817a4-kube-api-access-95xkw\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:56:23.700800 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:23.700720    4294 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/84edfcc3-1664-4518-a95e-7a1b3c1817a4-bundle\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:56:23.700800 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:23.700729    4294 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/84edfcc3-1664-4518-a95e-7a1b3c1817a4-util\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:56:24.403132 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:24.403098    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp" event={"ID":"84edfcc3-1664-4518-a95e-7a1b3c1817a4","Type":"ContainerDied","Data":"4e94b14622d27ac86cdf7e15bbe85795777fcf152441a797bd968a3162978f63"}
Nov 02 22:56:24.403288 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:24.403138    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xxjzp"
Nov 02 22:56:24.403288 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:24.403147    4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e94b14622d27ac86cdf7e15bbe85795777fcf152441a797bd968a3162978f63"
Nov 02 22:56:30.753398 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:30.753366    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:41.747461 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:56:41.747423    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 22:56:44.279889 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:44.279853    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:44.440686 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:44.440652    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:48.282282 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:48.282248    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:48.578700 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:48.578622    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:49.506962 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.506929    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:49.712960 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.712926    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"]
Nov 02 22:56:49.713237 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.713221    4294 cpu_manager.go:414] "containerMap: removing container" podUID="f3c88446-d04e-46cb-8f5b-61b76aac49b7" containerName="util"
Nov 02 22:56:49.713278 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.713241    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c88446-d04e-46cb-8f5b-61b76aac49b7" containerName="util"
Nov 02 22:56:49.713278 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.713251    4294 cpu_manager.go:414] "containerMap: removing container" podUID="f3c88446-d04e-46cb-8f5b-61b76aac49b7" containerName="pull"
Nov 02 22:56:49.713278 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.713256    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c88446-d04e-46cb-8f5b-61b76aac49b7" containerName="pull"
Nov 02 22:56:49.713278 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.713264    4294 cpu_manager.go:414] "containerMap: removing container" podUID="84edfcc3-1664-4518-a95e-7a1b3c1817a4" containerName="pull"
Nov 02 22:56:49.713278 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.713270    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84edfcc3-1664-4518-a95e-7a1b3c1817a4" containerName="pull"
Nov 02 22:56:49.713278 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.713276    4294 cpu_manager.go:414] "containerMap: removing container" podUID="84edfcc3-1664-4518-a95e-7a1b3c1817a4" containerName="extract"
Nov 02 22:56:49.713450 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.713280    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84edfcc3-1664-4518-a95e-7a1b3c1817a4" containerName="extract"
Nov 02 22:56:49.713450 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.713298    4294 cpu_manager.go:414] "containerMap: removing container" podUID="f3c88446-d04e-46cb-8f5b-61b76aac49b7" containerName="extract"
Nov 02 22:56:49.713450 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.713304    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c88446-d04e-46cb-8f5b-61b76aac49b7" containerName="extract"
Nov 02 22:56:49.713450 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.713315    4294 cpu_manager.go:414] "containerMap: removing container" podUID="84edfcc3-1664-4518-a95e-7a1b3c1817a4" containerName="util"
Nov 02 22:56:49.713450 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.713322    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84edfcc3-1664-4518-a95e-7a1b3c1817a4" containerName="util"
Nov 02 22:56:49.713450 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.713371    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="84edfcc3-1664-4518-a95e-7a1b3c1817a4" containerName="extract"
Nov 02 22:56:49.713450 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.713381    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3c88446-d04e-46cb-8f5b-61b76aac49b7" containerName="extract"
Nov 02 22:56:49.733798 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.733769    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:49.786495 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.786419    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqh8l\" (UniqueName: \"kubernetes.io/projected/b9c1a998-9a6a-40de-bc3b-5cf7ebf87349-kube-api-access-vqh8l\") pod \"opendatahub-operator-controller-manager-7858c78b8d-dvwpk\" (UID: \"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:49.786495 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.786489    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9c1a998-9a6a-40de-bc3b-5cf7ebf87349-metrics-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-dvwpk\" (UID: \"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:49.786705 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.786530    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b9c1a998-9a6a-40de-bc3b-5cf7ebf87349-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-dvwpk\" (UID: \"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:49.786705 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.786596    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9c1a998-9a6a-40de-bc3b-5cf7ebf87349-webhook-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-dvwpk\" (UID: \"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:49.786705 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.786625    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9c1a998-9a6a-40de-bc3b-5cf7ebf87349-tmp\") pod \"opendatahub-operator-controller-manager-7858c78b8d-dvwpk\" (UID: \"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:49.786705 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.786681    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b9c1a998-9a6a-40de-bc3b-5cf7ebf87349-manifests\") pod \"opendatahub-operator-controller-manager-7858c78b8d-dvwpk\" (UID: \"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:49.803897 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.803869    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:49.834307 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.834279    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"]
Nov 02 22:56:49.845395 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.845370    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 22:56:49.845532 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.845518    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 22:56:49.845675 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.845658    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-lpc6c\""
Nov 02 22:56:49.845737 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.845661    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 22:56:49.845737 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.845702    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 22:56:49.845868 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.845854    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 22:56:49.888297 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.888158    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9c1a998-9a6a-40de-bc3b-5cf7ebf87349-metrics-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-dvwpk\" (UID: \"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:49.888297 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.888218    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b9c1a998-9a6a-40de-bc3b-5cf7ebf87349-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-dvwpk\" (UID: \"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:49.888297 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.888262    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9c1a998-9a6a-40de-bc3b-5cf7ebf87349-webhook-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-dvwpk\" (UID: \"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:49.888595 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.888386    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9c1a998-9a6a-40de-bc3b-5cf7ebf87349-tmp\") pod \"opendatahub-operator-controller-manager-7858c78b8d-dvwpk\" (UID: \"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:49.888595 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.888436    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b9c1a998-9a6a-40de-bc3b-5cf7ebf87349-manifests\") pod \"opendatahub-operator-controller-manager-7858c78b8d-dvwpk\" (UID: \"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:49.888595 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.888485    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqh8l\" (UniqueName: \"kubernetes.io/projected/b9c1a998-9a6a-40de-bc3b-5cf7ebf87349-kube-api-access-vqh8l\") pod \"opendatahub-operator-controller-manager-7858c78b8d-dvwpk\" (UID: \"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:49.889167 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.889139    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b9c1a998-9a6a-40de-bc3b-5cf7ebf87349-manifests\") pod \"opendatahub-operator-controller-manager-7858c78b8d-dvwpk\" (UID: \"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:49.890741 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.890715    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9c1a998-9a6a-40de-bc3b-5cf7ebf87349-tmp\") pod \"opendatahub-operator-controller-manager-7858c78b8d-dvwpk\" (UID: \"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:49.890887 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.890868    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b9c1a998-9a6a-40de-bc3b-5cf7ebf87349-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-dvwpk\" (UID: \"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:49.891044 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.891029    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9c1a998-9a6a-40de-bc3b-5cf7ebf87349-metrics-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-dvwpk\" (UID: \"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:49.891089 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.891057    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9c1a998-9a6a-40de-bc3b-5cf7ebf87349-webhook-cert\") pod \"opendatahub-operator-controller-manager-7858c78b8d-dvwpk\" (UID: \"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:49.943822 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.943789    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqh8l\" (UniqueName: \"kubernetes.io/projected/b9c1a998-9a6a-40de-bc3b-5cf7ebf87349-kube-api-access-vqh8l\") pod \"opendatahub-operator-controller-manager-7858c78b8d-dvwpk\" (UID: \"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:49.991444 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:49.991415    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:50.073245 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:50.073169    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:50.268924 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:50.268887    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:50.602611 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:50.602571    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"]
Nov 02 22:56:50.604505 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:56:50.604472    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c1a998_9a6a_40de_bc3b_5cf7ebf87349.slice/crio-e48b2f5fbb1284a1ac10b02b05cac5cf8e453f1d4a8038e3752f43d9bd38b69f WatchSource:0}: Error finding container e48b2f5fbb1284a1ac10b02b05cac5cf8e453f1d4a8038e3752f43d9bd38b69f: Status 404 returned error can't find the container with id e48b2f5fbb1284a1ac10b02b05cac5cf8e453f1d4a8038e3752f43d9bd38b69f
Nov 02 22:56:51.505311 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:51.505275    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk" event={"ID":"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349","Type":"ContainerStarted","Data":"e48b2f5fbb1284a1ac10b02b05cac5cf8e453f1d4a8038e3752f43d9bd38b69f"}
Nov 02 22:56:53.103120 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:53.103087    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:55.843560 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:55.843530    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:56.526701 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:56.526610    4294 generic.go:356] "Generic (PLEG): container finished" podID="b9c1a998-9a6a-40de-bc3b-5cf7ebf87349" containerID="0cdf4c4b5f24ac16490e7b0b285e839544af45d308819157158a713100472bd6" exitCode=0
Nov 02 22:56:56.526701 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:56.526649    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk" event={"ID":"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349","Type":"ContainerDied","Data":"0cdf4c4b5f24ac16490e7b0b285e839544af45d308819157158a713100472bd6"}
Nov 02 22:56:57.474413 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:57.474361    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:56:57.531561 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:57.531527    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk" event={"ID":"b9c1a998-9a6a-40de-bc3b-5cf7ebf87349","Type":"ContainerStarted","Data":"8200a8f1b2ef8db6de72ee383bbeb7619626c995ed0b9f18a2fd3117b6367553"}
Nov 02 22:56:57.531715 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:57.531622    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:56:57.756804 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:57.756753    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk" podStartSLOduration=2.271244267 podStartE2EDuration="8.756738108s" totalImagesPullingTime="6.02811953s" totalInitContainerRuntime="457.374311ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:56:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:56:57.708052643 +0000 UTC m=+257.276407432" watchObservedRunningTime="2026-11-02 22:56:57.756738108 +0000 UTC m=+257.325092895"
Nov 02 22:56:58.677442 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:56:58.677408    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:00.137453 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:00.137418    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:01.604643 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:01.604611    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:03.716176 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:03.716140    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:03.996557 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:03.996516    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:04.894783 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:04.894746    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:05.475115 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:05.475077    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:07.751437 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:07.751389    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:07.970418 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:07.970364    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:08.301869 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:08.301819    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:08.521669 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:08.521634    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:08.537680 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:08.537649    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7858c78b8d-dvwpk"
Nov 02 22:57:10.086441 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:10.086402    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:10.395204 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:10.395073    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:11.127727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:11.127681    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:11.524808 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:11.524772    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:11.747152 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:57:11.747113    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 22:57:13.705106 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:13.705052    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:13.880958 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:13.880885    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:15.829093 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:15.829061    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:16.024049 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:16.024016    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:17.673522 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:17.673487    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:17.840000 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:17.839950    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:18.186017 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:18.185959    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:18.358540 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:18.358504    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:19.672044 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:19.672005    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:20.008050 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:20.008014    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lpzm9"]
Nov 02 22:57:20.011226 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:20.011208    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lpzm9"
Nov 02 22:57:20.067830 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:20.067799    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 02 22:57:20.067830 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:20.067800    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-2mrvd\""
Nov 02 22:57:20.114571 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:20.114517    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 02 22:57:20.116066 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:20.116043    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lpzm9"]
Nov 02 22:57:20.146088 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:20.146054    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh75t\" (UniqueName: \"kubernetes.io/projected/73f000fe-e6cc-4650-ba3e-6432d4ad82c7-kube-api-access-xh75t\") pod \"cert-manager-cainjector-5c57bbd4fc-lpzm9\" (UID: \"73f000fe-e6cc-4650-ba3e-6432d4ad82c7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lpzm9"
Nov 02 22:57:20.146257 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:20.146120    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73f000fe-e6cc-4650-ba3e-6432d4ad82c7-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lpzm9\" (UID: \"73f000fe-e6cc-4650-ba3e-6432d4ad82c7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lpzm9"
Nov 02 22:57:20.246817 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:20.246775    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73f000fe-e6cc-4650-ba3e-6432d4ad82c7-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lpzm9\" (UID: \"73f000fe-e6cc-4650-ba3e-6432d4ad82c7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lpzm9"
Nov 02 22:57:20.247064 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:20.246841    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh75t\" (UniqueName: \"kubernetes.io/projected/73f000fe-e6cc-4650-ba3e-6432d4ad82c7-kube-api-access-xh75t\") pod \"cert-manager-cainjector-5c57bbd4fc-lpzm9\" (UID: \"73f000fe-e6cc-4650-ba3e-6432d4ad82c7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lpzm9"
Nov 02 22:57:20.304140 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:20.304068    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73f000fe-e6cc-4650-ba3e-6432d4ad82c7-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lpzm9\" (UID: \"73f000fe-e6cc-4650-ba3e-6432d4ad82c7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lpzm9"
Nov 02 22:57:20.304259 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:20.304184    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh75t\" (UniqueName: \"kubernetes.io/projected/73f000fe-e6cc-4650-ba3e-6432d4ad82c7-kube-api-access-xh75t\") pod \"cert-manager-cainjector-5c57bbd4fc-lpzm9\" (UID: \"73f000fe-e6cc-4650-ba3e-6432d4ad82c7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lpzm9"
Nov 02 22:57:20.335490 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:20.335455    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lpzm9"
Nov 02 22:57:20.694908 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:20.694881    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lpzm9"]
Nov 02 22:57:20.696763 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:57:20.696734    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73f000fe_e6cc_4650_ba3e_6432d4ad82c7.slice/crio-454b55eaef1cf28dfa3bedda3eef2bd44b080bbdb0c73edfedd1fcb607cfa240 WatchSource:0}: Error finding container 454b55eaef1cf28dfa3bedda3eef2bd44b080bbdb0c73edfedd1fcb607cfa240: Status 404 returned error can't find the container with id 454b55eaef1cf28dfa3bedda3eef2bd44b080bbdb0c73edfedd1fcb607cfa240
Nov 02 22:57:21.617163 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:21.617124    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lpzm9" event={"ID":"73f000fe-e6cc-4650-ba3e-6432d4ad82c7","Type":"ContainerStarted","Data":"454b55eaef1cf28dfa3bedda3eef2bd44b080bbdb0c73edfedd1fcb607cfa240"}
Nov 02 22:57:22.413829 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:22.413787    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:22.587545 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:22.587507    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:22.752901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:22.752865    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:23.580202 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:23.580172    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:23.752192 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:23.752153    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:23.911994 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:23.911895    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:24.284477 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:24.284441    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:24.444228 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:24.444196    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:24.969187 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:24.969152    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:25.146258 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:25.146219    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:25.473572 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:25.473537    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:25.637459 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:25.637361    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lpzm9" event={"ID":"73f000fe-e6cc-4650-ba3e-6432d4ad82c7","Type":"ContainerStarted","Data":"c2bbc2843b802e76803c3ed5933022748c7d4c6f93ba00368938feb9ef8b44cf"}
Nov 02 22:57:25.747123 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:25.747062    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lpzm9" podStartSLOduration=2.187448173 podStartE2EDuration="6.747047978s" totalImagesPullingTime="4.559599805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:25.746298819 +0000 UTC m=+285.314653607" watchObservedRunningTime="2026-11-02 22:57:25.747047978 +0000 UTC m=+285.315402762"
Nov 02 22:57:26.861638 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:26.861600    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:27.009275 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.009235    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7cc54b748b-wp6k5"]
Nov 02 22:57:27.012708 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.012686    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cc54b748b-wp6k5"
Nov 02 22:57:27.033110 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.033081    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:27.093814 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.093781    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cc54b748b-wp6k5"]
Nov 02 22:57:27.097255 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.097226    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0db34b7-c04a-4f18-aa19-725f7870c7ed-tls-certs\") pod \"kube-auth-proxy-7cc54b748b-wp6k5\" (UID: \"b0db34b7-c04a-4f18-aa19-725f7870c7ed\") " pod="openshift-ingress/kube-auth-proxy-7cc54b748b-wp6k5"
Nov 02 22:57:27.097407 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.097326    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0db34b7-c04a-4f18-aa19-725f7870c7ed-tmp\") pod \"kube-auth-proxy-7cc54b748b-wp6k5\" (UID: \"b0db34b7-c04a-4f18-aa19-725f7870c7ed\") " pod="openshift-ingress/kube-auth-proxy-7cc54b748b-wp6k5"
Nov 02 22:57:27.097407 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.097360    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmgh8\" (UniqueName: \"kubernetes.io/projected/b0db34b7-c04a-4f18-aa19-725f7870c7ed-kube-api-access-pmgh8\") pod \"kube-auth-proxy-7cc54b748b-wp6k5\" (UID: \"b0db34b7-c04a-4f18-aa19-725f7870c7ed\") " pod="openshift-ingress/kube-auth-proxy-7cc54b748b-wp6k5"
Nov 02 22:57:27.120197 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.120119    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 22:57:27.120197 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.120109    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 22:57:27.120365 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.120207    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jx5r7\""
Nov 02 22:57:27.198271 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.198243    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0db34b7-c04a-4f18-aa19-725f7870c7ed-tmp\") pod \"kube-auth-proxy-7cc54b748b-wp6k5\" (UID: \"b0db34b7-c04a-4f18-aa19-725f7870c7ed\") " pod="openshift-ingress/kube-auth-proxy-7cc54b748b-wp6k5"
Nov 02 22:57:27.198428 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.198282    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmgh8\" (UniqueName: \"kubernetes.io/projected/b0db34b7-c04a-4f18-aa19-725f7870c7ed-kube-api-access-pmgh8\") pod \"kube-auth-proxy-7cc54b748b-wp6k5\" (UID: \"b0db34b7-c04a-4f18-aa19-725f7870c7ed\") " pod="openshift-ingress/kube-auth-proxy-7cc54b748b-wp6k5"
Nov 02 22:57:27.198428 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.198331    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0db34b7-c04a-4f18-aa19-725f7870c7ed-tls-certs\") pod \"kube-auth-proxy-7cc54b748b-wp6k5\" (UID: \"b0db34b7-c04a-4f18-aa19-725f7870c7ed\") " pod="openshift-ingress/kube-auth-proxy-7cc54b748b-wp6k5"
Nov 02 22:57:27.200694 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.200664    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0db34b7-c04a-4f18-aa19-725f7870c7ed-tmp\") pod \"kube-auth-proxy-7cc54b748b-wp6k5\" (UID: \"b0db34b7-c04a-4f18-aa19-725f7870c7ed\") " pod="openshift-ingress/kube-auth-proxy-7cc54b748b-wp6k5"
Nov 02 22:57:27.201011 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.200973    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0db34b7-c04a-4f18-aa19-725f7870c7ed-tls-certs\") pod \"kube-auth-proxy-7cc54b748b-wp6k5\" (UID: \"b0db34b7-c04a-4f18-aa19-725f7870c7ed\") " pod="openshift-ingress/kube-auth-proxy-7cc54b748b-wp6k5"
Nov 02 22:57:27.209273 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.209243    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:27.256540 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.256513    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmgh8\" (UniqueName: \"kubernetes.io/projected/b0db34b7-c04a-4f18-aa19-725f7870c7ed-kube-api-access-pmgh8\") pod \"kube-auth-proxy-7cc54b748b-wp6k5\" (UID: \"b0db34b7-c04a-4f18-aa19-725f7870c7ed\") " pod="openshift-ingress/kube-auth-proxy-7cc54b748b-wp6k5"
Nov 02 22:57:27.323579 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.323547    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cc54b748b-wp6k5"
Nov 02 22:57:27.449632 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.449599    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:27.673551 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.673348    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:27.693477 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.693452    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cc54b748b-wp6k5"]
Nov 02 22:57:27.695411 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:57:27.695385    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0db34b7_c04a_4f18_aa19_725f7870c7ed.slice/crio-b3cae27e669e680504987154990cf9d68b6f2fd2f7824fb596ccd705075768f0 WatchSource:0}: Error finding container b3cae27e669e680504987154990cf9d68b6f2fd2f7824fb596ccd705075768f0: Status 404 returned error can't find the container with id b3cae27e669e680504987154990cf9d68b6f2fd2f7824fb596ccd705075768f0
Nov 02 22:57:27.851873 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:27.851817    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:28.026296 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:28.026248    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:28.599203 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:28.599167    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:28.650172 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:28.650141    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cc54b748b-wp6k5" event={"ID":"b0db34b7-c04a-4f18-aa19-725f7870c7ed","Type":"ContainerStarted","Data":"b3cae27e669e680504987154990cf9d68b6f2fd2f7824fb596ccd705075768f0"}
Nov 02 22:57:28.768462 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:28.768423    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:30.428339 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:30.428307    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:30.726619 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:30.726543    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:31.663473 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:31.663435    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cc54b748b-wp6k5" event={"ID":"b0db34b7-c04a-4f18-aa19-725f7870c7ed","Type":"ContainerStarted","Data":"ec7078e2fccd42df36f274013416d8170b87dc9cd1414ec9748008bc15ea3c13"}
Nov 02 22:57:31.776081 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:31.776011    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7cc54b748b-wp6k5" podStartSLOduration=2.702944978 podStartE2EDuration="5.775993261s" totalImagesPullingTime="3.073048283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:31.774384515 +0000 UTC m=+291.342739302" watchObservedRunningTime="2026-11-02 22:57:31.775993261 +0000 UTC m=+291.344348040"
Nov 02 22:57:34.500812 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:34.500763    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:35.140938 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:35.140881    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:35.365855 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:35.365813    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:36.579927 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:36.579881    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:36.743206 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:36.743155    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:36.911847 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:36.911737    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:38.072702 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.072658    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z"]
Nov 02 22:57:38.076166 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.076143    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z"
Nov 02 22:57:38.174757 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.174722    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-t28bh\""
Nov 02 22:57:38.174972 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.174790    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 22:57:38.174972 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.174799    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 22:57:38.174972 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.174818    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 02 22:57:38.174972 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.174825    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 02 22:57:38.174972 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.174949    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 02 22:57:38.176414 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.176388    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z"]
Nov 02 22:57:38.194068 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.194039    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/feaac210-7d68-4031-9075-e1d779025d21-manager-config\") pod \"jobset-controller-manager-685ff49898-9vj7z\" (UID: \"feaac210-7d68-4031-9075-e1d779025d21\") " pod="openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z"
Nov 02 22:57:38.194188 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.194072    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/feaac210-7d68-4031-9075-e1d779025d21-cert\") pod \"jobset-controller-manager-685ff49898-9vj7z\" (UID: \"feaac210-7d68-4031-9075-e1d779025d21\") " pod="openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z"
Nov 02 22:57:38.194231 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.194199    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feaac210-7d68-4031-9075-e1d779025d21-metrics-certs\") pod \"jobset-controller-manager-685ff49898-9vj7z\" (UID: \"feaac210-7d68-4031-9075-e1d779025d21\") " pod="openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z"
Nov 02 22:57:38.194264 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.194238    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl8qm\" (UniqueName: \"kubernetes.io/projected/feaac210-7d68-4031-9075-e1d779025d21-kube-api-access-rl8qm\") pod \"jobset-controller-manager-685ff49898-9vj7z\" (UID: \"feaac210-7d68-4031-9075-e1d779025d21\") " pod="openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z"
Nov 02 22:57:38.294613 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.294576    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/feaac210-7d68-4031-9075-e1d779025d21-manager-config\") pod \"jobset-controller-manager-685ff49898-9vj7z\" (UID: \"feaac210-7d68-4031-9075-e1d779025d21\") " pod="openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z"
Nov 02 22:57:38.294613 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.294613    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/feaac210-7d68-4031-9075-e1d779025d21-cert\") pod \"jobset-controller-manager-685ff49898-9vj7z\" (UID: \"feaac210-7d68-4031-9075-e1d779025d21\") " pod="openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z"
Nov 02 22:57:38.294822 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.294686    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feaac210-7d68-4031-9075-e1d779025d21-metrics-certs\") pod \"jobset-controller-manager-685ff49898-9vj7z\" (UID: \"feaac210-7d68-4031-9075-e1d779025d21\") " pod="openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z"
Nov 02 22:57:38.294857 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.294817    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl8qm\" (UniqueName: \"kubernetes.io/projected/feaac210-7d68-4031-9075-e1d779025d21-kube-api-access-rl8qm\") pod \"jobset-controller-manager-685ff49898-9vj7z\" (UID: \"feaac210-7d68-4031-9075-e1d779025d21\") " pod="openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z"
Nov 02 22:57:38.295382 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.295362    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/feaac210-7d68-4031-9075-e1d779025d21-manager-config\") pod \"jobset-controller-manager-685ff49898-9vj7z\" (UID: \"feaac210-7d68-4031-9075-e1d779025d21\") " pod="openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z"
Nov 02 22:57:38.297143 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.297123    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/feaac210-7d68-4031-9075-e1d779025d21-metrics-certs\") pod \"jobset-controller-manager-685ff49898-9vj7z\" (UID: \"feaac210-7d68-4031-9075-e1d779025d21\") " pod="openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z"
Nov 02 22:57:38.297229 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.297186    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/feaac210-7d68-4031-9075-e1d779025d21-cert\") pod \"jobset-controller-manager-685ff49898-9vj7z\" (UID: \"feaac210-7d68-4031-9075-e1d779025d21\") " pod="openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z"
Nov 02 22:57:38.349459 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.349396    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl8qm\" (UniqueName: \"kubernetes.io/projected/feaac210-7d68-4031-9075-e1d779025d21-kube-api-access-rl8qm\") pod \"jobset-controller-manager-685ff49898-9vj7z\" (UID: \"feaac210-7d68-4031-9075-e1d779025d21\") " pod="openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z"
Nov 02 22:57:38.388158 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.388125    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z"
Nov 02 22:57:38.748018 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.747993    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z"]
Nov 02 22:57:38.750032 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:57:38.750006    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeaac210_7d68_4031_9075_e1d779025d21.slice/crio-da25c5d73758033c8b6816f9f96f10884818907d1ce38852abdc64aaf5a1a5f5 WatchSource:0}: Error finding container da25c5d73758033c8b6816f9f96f10884818907d1ce38852abdc64aaf5a1a5f5: Status 404 returned error can't find the container with id da25c5d73758033c8b6816f9f96f10884818907d1ce38852abdc64aaf5a1a5f5
Nov 02 22:57:38.886905 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:38.886866    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:39.090066 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:39.089964    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:39.503451 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:39.503413    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:39.654575 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:39.654541    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:39.695025 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:39.694969    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z" event={"ID":"feaac210-7d68-4031-9075-e1d779025d21","Type":"ContainerStarted","Data":"da25c5d73758033c8b6816f9f96f10884818907d1ce38852abdc64aaf5a1a5f5"}
Nov 02 22:57:41.751992 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:57:41.751942    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 22:57:42.559661 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:42.559620    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:43.713793 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:43.713755    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z" event={"ID":"feaac210-7d68-4031-9075-e1d779025d21","Type":"ContainerStarted","Data":"c64c8e51f8698c23c52e710b9bff804d1d4c884692370e321b1d720540d18542"}
Nov 02 22:57:43.714321 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:43.713888    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z"
Nov 02 22:57:43.825250 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:43.825189    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z" podStartSLOduration=1.486323392 podStartE2EDuration="5.825175524s" totalImagesPullingTime="4.338852132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:57:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:57:43.823778855 +0000 UTC m=+303.392133642" watchObservedRunningTime="2026-11-02 22:57:43.825175524 +0000 UTC m=+303.393530311"
Nov 02 22:57:43.908088 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:43.908056    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:44.073372 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:44.073336    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:44.351096 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:44.351015    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:44.513846 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:44.513815    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:44.695090 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:44.695012    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:44.863081 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:44.863047    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:46.799365 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:46.799327    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:46.965368 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:46.965333    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:50.161545 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:50.161508    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:50.323021 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:50.322963    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:54.723368 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:54.723294    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-685ff49898-9vj7z"
Nov 02 22:57:54.981705 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:54.981631    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:57:55.144329 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:57:55.144296    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:07.744637 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:07.744570    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:07.916353 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:07.916319    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:08.494085 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:08.494048    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:11.748151 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:58:11.748104    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 22:58:18.679028 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:18.678973    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:18.835742 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:18.835709    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:21.728512 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:21.728477    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:21.895903 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:21.895868    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:22.750631 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:22.750593    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:29.568107 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:29.568055    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:29.740771 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:29.740738    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:29.929303 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:29.929224    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:30.088000 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:30.087948    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:37.309569 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:37.309538    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:37.482245 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:37.482214    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:40.245315 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:40.245279    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:40.560563 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:40.560489    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:41.748872 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:58:41.748833    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 22:58:58.924938 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:58.924899    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:59.092066 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:59.092034    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:59.257883 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:59.257847    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:59.416133 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:59.416100    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:59.582172 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:59.582088    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:59.745974 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:59.745939    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:58:59.924648 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:58:59.924561    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:59:00.082881 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.082852    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:59:00.117725 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.117693    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"]
Nov 02 22:59:00.117901 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.117881    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9" podUID="35db14cd-79a6-46bd-93cb-2368586e9ccb" containerName="discovery" containerID="cri-o://7e774fcdf9463be473650b69fcac7ed9ae6d05eaf4695dd56607a3af87176be5" gracePeriod=30
Nov 02 22:59:00.165278 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.165241    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-4njxm"]
Nov 02 22:59:00.168332 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.168312    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-4njxm"
Nov 02 22:59:00.219607 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.219577    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 22:59:00.219607 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.219577    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 22:59:00.221165 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.221136    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-4njxm"]
Nov 02 22:59:00.276733 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.276706    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 02 22:59:00.276733 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.276711    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2nlb8\""
Nov 02 22:59:00.276733 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.276711    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 02 22:59:00.297614 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.297575    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b22dd01-17ba-4504-9208-7e3f35ff268b-metrics-certs\") pod \"odh-model-controller-769875db56-4njxm\" (UID: \"5b22dd01-17ba-4504-9208-7e3f35ff268b\") " pod="opendatahub/odh-model-controller-769875db56-4njxm"
Nov 02 22:59:00.297767 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.297624    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nqx9\" (UniqueName: \"kubernetes.io/projected/5b22dd01-17ba-4504-9208-7e3f35ff268b-kube-api-access-5nqx9\") pod \"odh-model-controller-769875db56-4njxm\" (UID: \"5b22dd01-17ba-4504-9208-7e3f35ff268b\") " pod="opendatahub/odh-model-controller-769875db56-4njxm"
Nov 02 22:59:00.297851 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.297833    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b22dd01-17ba-4504-9208-7e3f35ff268b-cert\") pod \"odh-model-controller-769875db56-4njxm\" (UID: \"5b22dd01-17ba-4504-9208-7e3f35ff268b\") " pod="opendatahub/odh-model-controller-769875db56-4njxm"
Nov 02 22:59:00.365042 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.365013    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:59:00.398596 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.398555    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b22dd01-17ba-4504-9208-7e3f35ff268b-metrics-certs\") pod \"odh-model-controller-769875db56-4njxm\" (UID: \"5b22dd01-17ba-4504-9208-7e3f35ff268b\") " pod="opendatahub/odh-model-controller-769875db56-4njxm"
Nov 02 22:59:00.398596 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.398598    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nqx9\" (UniqueName: \"kubernetes.io/projected/5b22dd01-17ba-4504-9208-7e3f35ff268b-kube-api-access-5nqx9\") pod \"odh-model-controller-769875db56-4njxm\" (UID: \"5b22dd01-17ba-4504-9208-7e3f35ff268b\") " pod="opendatahub/odh-model-controller-769875db56-4njxm"
Nov 02 22:59:00.398847 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.398697    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b22dd01-17ba-4504-9208-7e3f35ff268b-cert\") pod \"odh-model-controller-769875db56-4njxm\" (UID: \"5b22dd01-17ba-4504-9208-7e3f35ff268b\") " pod="opendatahub/odh-model-controller-769875db56-4njxm"
Nov 02 22:59:00.401038 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.401006    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b22dd01-17ba-4504-9208-7e3f35ff268b-metrics-certs\") pod \"odh-model-controller-769875db56-4njxm\" (UID: \"5b22dd01-17ba-4504-9208-7e3f35ff268b\") " pod="opendatahub/odh-model-controller-769875db56-4njxm"
Nov 02 22:59:00.401183 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.401162    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b22dd01-17ba-4504-9208-7e3f35ff268b-cert\") pod \"odh-model-controller-769875db56-4njxm\" (UID: \"5b22dd01-17ba-4504-9208-7e3f35ff268b\") " pod="opendatahub/odh-model-controller-769875db56-4njxm"
Nov 02 22:59:00.453149 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.453076    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nqx9\" (UniqueName: \"kubernetes.io/projected/5b22dd01-17ba-4504-9208-7e3f35ff268b-kube-api-access-5nqx9\") pod \"odh-model-controller-769875db56-4njxm\" (UID: \"5b22dd01-17ba-4504-9208-7e3f35ff268b\") " pod="opendatahub/odh-model-controller-769875db56-4njxm"
Nov 02 22:59:00.481057 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.481020    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-4njxm"
Nov 02 22:59:00.499480 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.499449    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-token\" (UniqueName: \"kubernetes.io/projected/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-token\") pod \"35db14cd-79a6-46bd-93cb-2368586e9ccb\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") "
Nov 02 22:59:00.499648 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.499491    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-cacerts\" (UniqueName: \"kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-cacerts\") pod \"35db14cd-79a6-46bd-93cb-2368586e9ccb\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") "
Nov 02 22:59:00.499648 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.499533    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-csr-dns-cert\") pod \"35db14cd-79a6-46bd-93cb-2368586e9ccb\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") "
Nov 02 22:59:00.499648 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.499598    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35db14cd-79a6-46bd-93cb-2368586e9ccb-kube-api-access-z8zvp\" (UniqueName: \"kubernetes.io/projected/35db14cd-79a6-46bd-93cb-2368586e9ccb-kube-api-access-z8zvp\") pod \"35db14cd-79a6-46bd-93cb-2368586e9ccb\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") "
Nov 02 22:59:00.499648 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.499638    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-kubeconfig\") pod \"35db14cd-79a6-46bd-93cb-2368586e9ccb\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") "
Nov 02 22:59:00.499867 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.499657    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35db14cd-79a6-46bd-93cb-2368586e9ccb-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/35db14cd-79a6-46bd-93cb-2368586e9ccb-local-certs\") pod \"35db14cd-79a6-46bd-93cb-2368586e9ccb\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") "
Nov 02 22:59:00.499867 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.499675    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-csr-ca-configmap\") pod \"35db14cd-79a6-46bd-93cb-2368586e9ccb\" (UID: \"35db14cd-79a6-46bd-93cb-2368586e9ccb\") "
Nov 02 22:59:00.500359 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.500332    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-csr-ca-configmap" pod "35db14cd-79a6-46bd-93cb-2368586e9ccb" (UID: "35db14cd-79a6-46bd-93cb-2368586e9ccb"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:59:00.500599 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.500582    4294 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-csr-ca-configmap\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:59:00.502516 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.502478    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-cacerts" pod "35db14cd-79a6-46bd-93cb-2368586e9ccb" (UID: "35db14cd-79a6-46bd-93cb-2368586e9ccb"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:59:00.502608 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.502517    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-kubeconfig" pod "35db14cd-79a6-46bd-93cb-2368586e9ccb" (UID: "35db14cd-79a6-46bd-93cb-2368586e9ccb"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:59:00.502608 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.502540    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-csr-dns-cert" pod "35db14cd-79a6-46bd-93cb-2368586e9ccb" (UID: "35db14cd-79a6-46bd-93cb-2368586e9ccb"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:59:00.502608 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.502554    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35db14cd-79a6-46bd-93cb-2368586e9ccb-kube-api-access-z8zvp" pod "35db14cd-79a6-46bd-93cb-2368586e9ccb" (UID: "35db14cd-79a6-46bd-93cb-2368586e9ccb"). InnerVolumeSpecName "kube-api-access-z8zvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:59:00.502608 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.502540    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-token" pod "35db14cd-79a6-46bd-93cb-2368586e9ccb" (UID: "35db14cd-79a6-46bd-93cb-2368586e9ccb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:59:00.502864 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.502841    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35db14cd-79a6-46bd-93cb-2368586e9ccb-local-certs" pod "35db14cd-79a6-46bd-93cb-2368586e9ccb" (UID: "35db14cd-79a6-46bd-93cb-2368586e9ccb"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 22:59:00.601333 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.601297    4294 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-kubeconfig\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:59:00.601333 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.601327    4294 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/35db14cd-79a6-46bd-93cb-2368586e9ccb-local-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:59:00.601333 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.601336    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:59:00.601566 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.601345    4294 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-cacerts\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:59:00.601566 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.601355    4294 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/35db14cd-79a6-46bd-93cb-2368586e9ccb-istio-csr-dns-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:59:00.601566 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.601363    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8zvp\" (UniqueName: \"kubernetes.io/projected/35db14cd-79a6-46bd-93cb-2368586e9ccb-kube-api-access-z8zvp\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:59:00.848091 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.848063    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-4njxm"]
Nov 02 22:59:00.849748 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:59:00.849720    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b22dd01_17ba_4504_9208_7e3f35ff268b.slice/crio-8dd39e7ff68288b07394724aed9d0eba2879d7e65321393460ca0f5bf7cdb03c WatchSource:0}: Error finding container 8dd39e7ff68288b07394724aed9d0eba2879d7e65321393460ca0f5bf7cdb03c: Status 404 returned error can't find the container with id 8dd39e7ff68288b07394724aed9d0eba2879d7e65321393460ca0f5bf7cdb03c
Nov 02 22:59:00.851027 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:00.851007    4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 22:59:01.022945 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:01.022905    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-4njxm" event={"ID":"5b22dd01-17ba-4504-9208-7e3f35ff268b","Type":"ContainerStarted","Data":"8dd39e7ff68288b07394724aed9d0eba2879d7e65321393460ca0f5bf7cdb03c"}
Nov 02 22:59:01.024209 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:01.024181    4294 generic.go:356] "Generic (PLEG): container finished" podID="35db14cd-79a6-46bd-93cb-2368586e9ccb" containerID="7e774fcdf9463be473650b69fcac7ed9ae6d05eaf4695dd56607a3af87176be5" exitCode=0
Nov 02 22:59:01.024329 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:01.024231    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"
Nov 02 22:59:01.024329 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:01.024237    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9" event={"ID":"35db14cd-79a6-46bd-93cb-2368586e9ccb","Type":"ContainerDied","Data":"7e774fcdf9463be473650b69fcac7ed9ae6d05eaf4695dd56607a3af87176be5"}
Nov 02 22:59:01.024329 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:01.024271    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9" event={"ID":"35db14cd-79a6-46bd-93cb-2368586e9ccb","Type":"ContainerDied","Data":"b2bec09fcfdcaea9ee1016dc39ebaa88eee76b5fb451641f7c900976f8365e23"}
Nov 02 22:59:01.024329 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:01.024290    4294 scope.go:122] "RemoveContainer" containerID="7e774fcdf9463be473650b69fcac7ed9ae6d05eaf4695dd56607a3af87176be5"
Nov 02 22:59:01.032562 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:01.032541    4294 scope.go:122] "RemoveContainer" containerID="7e774fcdf9463be473650b69fcac7ed9ae6d05eaf4695dd56607a3af87176be5"
Nov 02 22:59:01.032857 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:59:01.032838    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e774fcdf9463be473650b69fcac7ed9ae6d05eaf4695dd56607a3af87176be5\": container with ID starting with 7e774fcdf9463be473650b69fcac7ed9ae6d05eaf4695dd56607a3af87176be5 not found: ID does not exist" containerID="7e774fcdf9463be473650b69fcac7ed9ae6d05eaf4695dd56607a3af87176be5"
Nov 02 22:59:01.032915 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:01.032871    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e774fcdf9463be473650b69fcac7ed9ae6d05eaf4695dd56607a3af87176be5"} err="failed to get container status \"7e774fcdf9463be473650b69fcac7ed9ae6d05eaf4695dd56607a3af87176be5\": rpc error: code = NotFound desc = could not find container \"7e774fcdf9463be473650b69fcac7ed9ae6d05eaf4695dd56607a3af87176be5\": container with ID starting with 7e774fcdf9463be473650b69fcac7ed9ae6d05eaf4695dd56607a3af87176be5 not found: ID does not exist"
Nov 02 22:59:01.183631 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:01.183553    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"]
Nov 02 22:59:01.189868 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:01.189834    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9"]
Nov 02 22:59:01.244533 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:01.244495    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:59:02.547101 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:02.547065    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35db14cd-79a6-46bd-93cb-2368586e9ccb" path="/var/lib/kubelet/pods/35db14cd-79a6-46bd-93cb-2368586e9ccb/volumes"
Nov 02 22:59:05.050261 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:05.050221    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-4njxm" event={"ID":"5b22dd01-17ba-4504-9208-7e3f35ff268b","Type":"ContainerStarted","Data":"bacc2f1ac22440cb34d79e863f5f7de4e3f69ddc736a8236a660e3fb14858f68"}
Nov 02 22:59:05.050673 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:05.050339    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-4njxm"
Nov 02 22:59:05.159155 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:05.159102    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-4njxm" podStartSLOduration=1.8214341269999998 podStartE2EDuration="5.159087805s" totalImagesPullingTime="3.337653678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:59:05.158238327 +0000 UTC m=+384.726593115" watchObservedRunningTime="2026-11-02 22:59:05.159087805 +0000 UTC m=+384.727442619"
Nov 02 22:59:05.329458 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:05.329347    4294 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-jfth9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.19:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 22:59:05.329458 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:05.329423    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jfth9" podUID="35db14cd-79a6-46bd-93cb-2368586e9ccb" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.19:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 22:59:06.530453 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:06.530422    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:59:06.687212 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:06.687177    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:59:06.846196 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:06.846111    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:59:07.013093 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:07.013058    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:59:07.172649 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:07.172563    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:59:07.329154 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:07.329127    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:59:07.489041 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:07.488925    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:59:07.648548 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:07.648513    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:59:07.821039 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:07.820998    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:59:10.499831 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:10.499795    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:59:10.674974 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:10.674940    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:59:11.748798 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:59:11.748760    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 22:59:12.367038 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:12.367000    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:59:12.534615 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:12.534566    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:59:12.690841 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:12.690737    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:59:13.388096 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.388060    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"]
Nov 02 22:59:13.388494 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.388370    4294 cpu_manager.go:414] "containerMap: removing container" podUID="35db14cd-79a6-46bd-93cb-2368586e9ccb" containerName="discovery"
Nov 02 22:59:13.388494 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.388381    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35db14cd-79a6-46bd-93cb-2368586e9ccb" containerName="discovery"
Nov 02 22:59:13.388494 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.388435    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="35db14cd-79a6-46bd-93cb-2368586e9ccb" containerName="discovery"
Nov 02 22:59:13.391479 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.391461    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"
Nov 02 22:59:13.417121 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.417084    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-metrics-certs\") pod \"maas-api-5ccf48f65-pbk4z\" (UID: \"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"
Nov 02 22:59:13.417277 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.417129    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc8ws\" (UniqueName: \"kubernetes.io/projected/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-kube-api-access-hc8ws\") pod \"maas-api-5ccf48f65-pbk4z\" (UID: \"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"
Nov 02 22:59:13.417277 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.417235    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-maas-api-tls\") pod \"maas-api-5ccf48f65-pbk4z\" (UID: \"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"
Nov 02 22:59:13.417277 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.417270    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-openshift-service-ca\") pod \"maas-api-5ccf48f65-pbk4z\" (UID: \"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"
Nov 02 22:59:13.490716 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.490670    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 22:59:13.490716 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.490698    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b7gfc\""
Nov 02 22:59:13.490716 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.490703    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 22:59:13.490970 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.490704    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 22:59:13.490970 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.490764    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 22:59:13.491931 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.491911    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"]
Nov 02 22:59:13.518481 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.518445    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-openshift-service-ca\") pod \"maas-api-5ccf48f65-pbk4z\" (UID: \"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"
Nov 02 22:59:13.518646 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.518508    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-metrics-certs\") pod \"maas-api-5ccf48f65-pbk4z\" (UID: \"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"
Nov 02 22:59:13.518646 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.518526    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc8ws\" (UniqueName: \"kubernetes.io/projected/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-kube-api-access-hc8ws\") pod \"maas-api-5ccf48f65-pbk4z\" (UID: \"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"
Nov 02 22:59:13.518646 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.518563    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-maas-api-tls\") pod \"maas-api-5ccf48f65-pbk4z\" (UID: \"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"
Nov 02 22:59:13.519294 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.519244    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-openshift-service-ca\") pod \"maas-api-5ccf48f65-pbk4z\" (UID: \"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"
Nov 02 22:59:13.520962 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.520943    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-maas-api-tls\") pod \"maas-api-5ccf48f65-pbk4z\" (UID: \"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"
Nov 02 22:59:13.521048 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.521030    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-metrics-certs\") pod \"maas-api-5ccf48f65-pbk4z\" (UID: \"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"
Nov 02 22:59:13.573288 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.573255    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc8ws\" (UniqueName: \"kubernetes.io/projected/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-kube-api-access-hc8ws\") pod \"maas-api-5ccf48f65-pbk4z\" (UID: \"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f\") " pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"
Nov 02 22:59:13.704212 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:13.704108    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"
Nov 02 22:59:14.067565 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:14.067539    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"]
Nov 02 22:59:14.069346 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:59:14.069323    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47339cf5_575d_4acf_b0f2_d4f64fe6ca3f.slice/crio-7f235fdb6d65676b86ee087b5f2b77588cbadcf117876fd274cffb53f6840e82 WatchSource:0}: Error finding container 7f235fdb6d65676b86ee087b5f2b77588cbadcf117876fd274cffb53f6840e82: Status 404 returned error can't find the container with id 7f235fdb6d65676b86ee087b5f2b77588cbadcf117876fd274cffb53f6840e82
Nov 02 22:59:14.087847 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:14.087810    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z" event={"ID":"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f","Type":"ContainerStarted","Data":"7f235fdb6d65676b86ee087b5f2b77588cbadcf117876fd274cffb53f6840e82"}
Nov 02 22:59:16.057478 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:16.057444    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-4njxm"
Nov 02 22:59:22.127624 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:22.127591    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z" event={"ID":"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f","Type":"ContainerStarted","Data":"db0c99bea4790b55009af4ee228775731009ec7c03f61355526701d06b5ad56d"}
Nov 02 22:59:22.128093 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:22.127667    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"
Nov 02 22:59:22.238344 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:22.238279    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z" podStartSLOduration=1.878834621 podStartE2EDuration="9.238261151s" totalImagesPullingTime="7.35942653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:59:22.236680712 +0000 UTC m=+401.805035499" watchObservedRunningTime="2026-11-02 22:59:22.238261151 +0000 UTC m=+401.806615940"
Nov 02 22:59:23.070526 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:23.070490    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:59:28.136872 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:28.136789    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"
Nov 02 22:59:28.651872 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:28.651838    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:59:28.894437 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:28.894406    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 22:59:28.911727 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:28.911648    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zm68f"]
Nov 02 22:59:28.913674 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:28.913659    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-zm68f"
Nov 02 22:59:28.965803 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:28.965767    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 22:59:28.966068 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:28.966010    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-p54bz\""
Nov 02 22:59:28.966139 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:28.966068    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 22:59:28.967517 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:28.967489    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zm68f"]
Nov 02 22:59:29.051383 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.051346    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjhqt\" (UniqueName: \"kubernetes.io/projected/556dcdc5-e48c-49d1-aa68-c1a25f938ccc-kube-api-access-bjhqt\") pod \"authorino-6f6944bcdd-zm68f\" (UID: \"556dcdc5-e48c-49d1-aa68-c1a25f938ccc\") " pod="kuadrant-system/authorino-6f6944bcdd-zm68f"
Nov 02 22:59:29.092291 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.092257    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"]
Nov 02 22:59:29.094486 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.094466    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"
Nov 02 22:59:29.150325 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.150296    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"]
Nov 02 22:59:29.151820 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.151785    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-metrics-certs\") pod \"maas-api-65f5668669-dkvfw\" (UID: \"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"
Nov 02 22:59:29.151957 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.151828    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mghwg\" (UniqueName: \"kubernetes.io/projected/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-kube-api-access-mghwg\") pod \"maas-api-65f5668669-dkvfw\" (UID: \"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"
Nov 02 22:59:29.152025 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.151962    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-maas-api-tls\") pod \"maas-api-65f5668669-dkvfw\" (UID: \"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"
Nov 02 22:59:29.152076 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.152041    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-openshift-service-ca\") pod \"maas-api-65f5668669-dkvfw\" (UID: \"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"
Nov 02 22:59:29.152125 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.152099    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjhqt\" (UniqueName: \"kubernetes.io/projected/556dcdc5-e48c-49d1-aa68-c1a25f938ccc-kube-api-access-bjhqt\") pod \"authorino-6f6944bcdd-zm68f\" (UID: \"556dcdc5-e48c-49d1-aa68-c1a25f938ccc\") " pod="kuadrant-system/authorino-6f6944bcdd-zm68f"
Nov 02 22:59:29.207012 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.206930    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjhqt\" (UniqueName: \"kubernetes.io/projected/556dcdc5-e48c-49d1-aa68-c1a25f938ccc-kube-api-access-bjhqt\") pod \"authorino-6f6944bcdd-zm68f\" (UID: \"556dcdc5-e48c-49d1-aa68-c1a25f938ccc\") " pod="kuadrant-system/authorino-6f6944bcdd-zm68f"
Nov 02 22:59:29.223779 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.223740    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-zm68f"
Nov 02 22:59:29.253332 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.253286    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-metrics-certs\") pod \"maas-api-65f5668669-dkvfw\" (UID: \"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"
Nov 02 22:59:29.253489 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.253339    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mghwg\" (UniqueName: \"kubernetes.io/projected/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-kube-api-access-mghwg\") pod \"maas-api-65f5668669-dkvfw\" (UID: \"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"
Nov 02 22:59:29.253489 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.253386    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-maas-api-tls\") pod \"maas-api-65f5668669-dkvfw\" (UID: \"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"
Nov 02 22:59:29.253489 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.253427    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-openshift-service-ca\") pod \"maas-api-65f5668669-dkvfw\" (UID: \"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"
Nov 02 22:59:29.254845 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.254809    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-openshift-service-ca\") pod \"maas-api-65f5668669-dkvfw\" (UID: \"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"
Nov 02 22:59:29.255744 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.255723    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-metrics-certs\") pod \"maas-api-65f5668669-dkvfw\" (UID: \"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"
Nov 02 22:59:29.255905 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.255885    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-maas-api-tls\") pod \"maas-api-65f5668669-dkvfw\" (UID: \"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"
Nov 02 22:59:29.261374 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.261342    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zm68f"]
Nov 02 22:59:29.282249 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.282215    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-69cf67b97b-t6vd6"]
Nov 02 22:59:29.294035 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.294006    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69cf67b97b-t6vd6"
Nov 02 22:59:29.307419 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.307393    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mghwg\" (UniqueName: \"kubernetes.io/projected/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-kube-api-access-mghwg\") pod \"maas-api-65f5668669-dkvfw\" (UID: \"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18\") " pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"
Nov 02 22:59:29.335555 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.335526    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69cf67b97b-t6vd6"]
Nov 02 22:59:29.354341 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.354301    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zjjc\" (UniqueName: \"kubernetes.io/projected/3a464ae8-add5-47bb-8716-cd3ae6805e4a-kube-api-access-2zjjc\") pod \"authorino-69cf67b97b-t6vd6\" (UID: \"3a464ae8-add5-47bb-8716-cd3ae6805e4a\") " pod="kuadrant-system/authorino-69cf67b97b-t6vd6"
Nov 02 22:59:29.405852 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.405822    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"
Nov 02 22:59:29.455538 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.455495    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zjjc\" (UniqueName: \"kubernetes.io/projected/3a464ae8-add5-47bb-8716-cd3ae6805e4a-kube-api-access-2zjjc\") pod \"authorino-69cf67b97b-t6vd6\" (UID: \"3a464ae8-add5-47bb-8716-cd3ae6805e4a\") " pod="kuadrant-system/authorino-69cf67b97b-t6vd6"
Nov 02 22:59:29.509906 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.509873    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zjjc\" (UniqueName: \"kubernetes.io/projected/3a464ae8-add5-47bb-8716-cd3ae6805e4a-kube-api-access-2zjjc\") pod \"authorino-69cf67b97b-t6vd6\" (UID: \"3a464ae8-add5-47bb-8716-cd3ae6805e4a\") " pod="kuadrant-system/authorino-69cf67b97b-t6vd6"
Nov 02 22:59:29.584288 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.584244    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zm68f"]
Nov 02 22:59:29.586397 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:59:29.586370    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod556dcdc5_e48c_49d1_aa68_c1a25f938ccc.slice/crio-3e7d74e374651e5bac9d528fddcde419d705471cac3dc11fb153451cb901cb56 WatchSource:0}: Error finding container 3e7d74e374651e5bac9d528fddcde419d705471cac3dc11fb153451cb901cb56: Status 404 returned error can't find the container with id 3e7d74e374651e5bac9d528fddcde419d705471cac3dc11fb153451cb901cb56
Nov 02 22:59:29.606544 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.606509    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69cf67b97b-t6vd6"
Nov 02 22:59:29.663868 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.663829    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69cf67b97b-t6vd6"]
Nov 02 22:59:29.685308 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.685280    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-588dc66bf4-jwdnm"]
Nov 02 22:59:29.687642 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.687624    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-588dc66bf4-jwdnm"
Nov 02 22:59:29.738237 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.738194    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 02 22:59:29.740073 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.740043    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-588dc66bf4-jwdnm"]
Nov 02 22:59:29.757610 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.757575    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44d4e4cc-09eb-4917-9626-a1a7c49935e9-tls-cert\") pod \"authorino-588dc66bf4-jwdnm\" (UID: \"44d4e4cc-09eb-4917-9626-a1a7c49935e9\") " pod="kuadrant-system/authorino-588dc66bf4-jwdnm"
Nov 02 22:59:29.757795 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.757694    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgq5n\" (UniqueName: \"kubernetes.io/projected/44d4e4cc-09eb-4917-9626-a1a7c49935e9-kube-api-access-pgq5n\") pod \"authorino-588dc66bf4-jwdnm\" (UID: \"44d4e4cc-09eb-4917-9626-a1a7c49935e9\") " pod="kuadrant-system/authorino-588dc66bf4-jwdnm"
Nov 02 22:59:29.772009 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.771971    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"]
Nov 02 22:59:29.773355 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:59:29.773324    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a47f445_7bff_4bed_9a6d_ba59d2e0fe18.slice/crio-ebd6df4842dc846e4e083e536e57a4548a8af3b431c0ecea3c06c77b37f1f9c2 WatchSource:0}: Error finding container ebd6df4842dc846e4e083e536e57a4548a8af3b431c0ecea3c06c77b37f1f9c2: Status 404 returned error can't find the container with id ebd6df4842dc846e4e083e536e57a4548a8af3b431c0ecea3c06c77b37f1f9c2
Nov 02 22:59:29.859103 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.859069    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgq5n\" (UniqueName: \"kubernetes.io/projected/44d4e4cc-09eb-4917-9626-a1a7c49935e9-kube-api-access-pgq5n\") pod \"authorino-588dc66bf4-jwdnm\" (UID: \"44d4e4cc-09eb-4917-9626-a1a7c49935e9\") " pod="kuadrant-system/authorino-588dc66bf4-jwdnm"
Nov 02 22:59:29.859103 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.859113    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44d4e4cc-09eb-4917-9626-a1a7c49935e9-tls-cert\") pod \"authorino-588dc66bf4-jwdnm\" (UID: \"44d4e4cc-09eb-4917-9626-a1a7c49935e9\") " pod="kuadrant-system/authorino-588dc66bf4-jwdnm"
Nov 02 22:59:29.861547 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.861519    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44d4e4cc-09eb-4917-9626-a1a7c49935e9-tls-cert\") pod \"authorino-588dc66bf4-jwdnm\" (UID: \"44d4e4cc-09eb-4917-9626-a1a7c49935e9\") " pod="kuadrant-system/authorino-588dc66bf4-jwdnm"
Nov 02 22:59:29.913930 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.913897    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgq5n\" (UniqueName: \"kubernetes.io/projected/44d4e4cc-09eb-4917-9626-a1a7c49935e9-kube-api-access-pgq5n\") pod \"authorino-588dc66bf4-jwdnm\" (UID: \"44d4e4cc-09eb-4917-9626-a1a7c49935e9\") " pod="kuadrant-system/authorino-588dc66bf4-jwdnm"
Nov 02 22:59:29.965488 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.965458    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69cf67b97b-t6vd6"]
Nov 02 22:59:29.968189 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:59:29.968163    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a464ae8_add5_47bb_8716_cd3ae6805e4a.slice/crio-3c1c5e0977574301b02320c2379e60e7f6a8fafdade039a6e4bba32ff77a61c2 WatchSource:0}: Error finding container 3c1c5e0977574301b02320c2379e60e7f6a8fafdade039a6e4bba32ff77a61c2: Status 404 returned error can't find the container with id 3c1c5e0977574301b02320c2379e60e7f6a8fafdade039a6e4bba32ff77a61c2
Nov 02 22:59:29.998539 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:29.998501    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-588dc66bf4-jwdnm"
Nov 02 22:59:30.160801 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:30.160764    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-zm68f" event={"ID":"556dcdc5-e48c-49d1-aa68-c1a25f938ccc","Type":"ContainerStarted","Data":"3e7d74e374651e5bac9d528fddcde419d705471cac3dc11fb153451cb901cb56"}
Nov 02 22:59:30.161872 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:30.161852    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69cf67b97b-t6vd6" event={"ID":"3a464ae8-add5-47bb-8716-cd3ae6805e4a","Type":"ContainerStarted","Data":"3c1c5e0977574301b02320c2379e60e7f6a8fafdade039a6e4bba32ff77a61c2"}
Nov 02 22:59:30.162860 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:30.162840    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw" event={"ID":"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18","Type":"ContainerStarted","Data":"ebd6df4842dc846e4e083e536e57a4548a8af3b431c0ecea3c06c77b37f1f9c2"}
Nov 02 22:59:30.361963 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:30.361936    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-588dc66bf4-jwdnm"]
Nov 02 22:59:30.363548 ip-10-0-139-234 kubenswrapper[4294]: W1102 22:59:30.363515    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44d4e4cc_09eb_4917_9626_a1a7c49935e9.slice/crio-b9ac1d2f31fcf7aa6be9e07f8978260dfa75c433f067c09a36bdabae50c223c5 WatchSource:0}: Error finding container b9ac1d2f31fcf7aa6be9e07f8978260dfa75c433f067c09a36bdabae50c223c5: Status 404 returned error can't find the container with id b9ac1d2f31fcf7aa6be9e07f8978260dfa75c433f067c09a36bdabae50c223c5
Nov 02 22:59:31.169275 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:31.169167    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw" event={"ID":"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18","Type":"ContainerStarted","Data":"1183abf138d4c4d5d3e17570644394779e6d86613c5a40f401fbee5a3a9aa368"}
Nov 02 22:59:31.169275 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:31.169270    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"
Nov 02 22:59:31.170326 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:31.170305    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-588dc66bf4-jwdnm" event={"ID":"44d4e4cc-09eb-4917-9626-a1a7c49935e9","Type":"ContainerStarted","Data":"b9ac1d2f31fcf7aa6be9e07f8978260dfa75c433f067c09a36bdabae50c223c5"}
Nov 02 22:59:31.278576 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:31.278524    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw" podStartSLOduration=1.257633528 podStartE2EDuration="2.278511321s" totalImagesPullingTime="1.020877793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:59:31.278054191 +0000 UTC m=+410.846408978" watchObservedRunningTime="2026-11-02 22:59:31.278511321 +0000 UTC m=+410.846866107"
Nov 02 22:59:35.195442 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:35.195402    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-zm68f" event={"ID":"556dcdc5-e48c-49d1-aa68-c1a25f938ccc","Type":"ContainerStarted","Data":"01dab2dd50cc331ae75d166da9c6b8344f61fa65e9b0ee069da2f660a6b6ca8f"}
Nov 02 22:59:35.195442 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:35.195439    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-zm68f" podUID="556dcdc5-e48c-49d1-aa68-c1a25f938ccc" containerName="authorino" containerID="cri-o://01dab2dd50cc331ae75d166da9c6b8344f61fa65e9b0ee069da2f660a6b6ca8f" gracePeriod=30
Nov 02 22:59:35.197112 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:35.197083    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69cf67b97b-t6vd6" event={"ID":"3a464ae8-add5-47bb-8716-cd3ae6805e4a","Type":"ContainerStarted","Data":"b56a7635a57a04bb5784d51dda219c7890879aee47172f615973b4f5c67b3196"}
Nov 02 22:59:35.197232 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:35.197160    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-69cf67b97b-t6vd6" podUID="3a464ae8-add5-47bb-8716-cd3ae6805e4a" containerName="authorino" containerID="cri-o://b56a7635a57a04bb5784d51dda219c7890879aee47172f615973b4f5c67b3196" gracePeriod=30
Nov 02 22:59:35.199062 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:35.199031    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-588dc66bf4-jwdnm" event={"ID":"44d4e4cc-09eb-4917-9626-a1a7c49935e9","Type":"ContainerStarted","Data":"a5c3d44f8322d84bdaae05fef221afa55500c80d354b4167e1ea534e208da3d2"}
Nov 02 22:59:35.302793 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:35.302735    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-zm68f" podStartSLOduration=2.385624373 podStartE2EDuration="7.302717009s" totalImagesPullingTime="4.917092636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:59:35.302451975 +0000 UTC m=+414.870806760" watchObservedRunningTime="2026-11-02 22:59:35.302717009 +0000 UTC m=+414.871071837"
Nov 02 22:59:35.411281 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:35.411204    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-588dc66bf4-jwdnm" podStartSLOduration=2.017880055 podStartE2EDuration="6.411183234s" totalImagesPullingTime="4.393303179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:59:35.410358732 +0000 UTC m=+414.978713522" watchObservedRunningTime="2026-11-02 22:59:35.411183234 +0000 UTC m=+414.979538022"
Nov 02 22:59:35.465666 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:35.465642    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69cf67b97b-t6vd6"
Nov 02 22:59:35.474405 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:35.474374    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-zm68f"
Nov 02 22:59:35.520042 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:35.519953    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-69cf67b97b-t6vd6" podStartSLOduration=1.935007653 podStartE2EDuration="6.519933197s" totalImagesPullingTime="4.584925544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:59:35.517897834 +0000 UTC m=+415.086252621" watchObservedRunningTime="2026-11-02 22:59:35.519933197 +0000 UTC m=+415.088287985"
Nov 02 22:59:35.611635 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:35.611590    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/556dcdc5-e48c-49d1-aa68-c1a25f938ccc-kube-api-access-bjhqt\" (UniqueName: \"kubernetes.io/projected/556dcdc5-e48c-49d1-aa68-c1a25f938ccc-kube-api-access-bjhqt\") pod \"556dcdc5-e48c-49d1-aa68-c1a25f938ccc\" (UID: \"556dcdc5-e48c-49d1-aa68-c1a25f938ccc\") "
Nov 02 22:59:35.611841 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:35.611651    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a464ae8-add5-47bb-8716-cd3ae6805e4a-kube-api-access-2zjjc\" (UniqueName: \"kubernetes.io/projected/3a464ae8-add5-47bb-8716-cd3ae6805e4a-kube-api-access-2zjjc\") pod \"3a464ae8-add5-47bb-8716-cd3ae6805e4a\" (UID: \"3a464ae8-add5-47bb-8716-cd3ae6805e4a\") "
Nov 02 22:59:35.613974 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:35.613935    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a464ae8-add5-47bb-8716-cd3ae6805e4a-kube-api-access-2zjjc" pod "3a464ae8-add5-47bb-8716-cd3ae6805e4a" (UID: "3a464ae8-add5-47bb-8716-cd3ae6805e4a"). InnerVolumeSpecName "kube-api-access-2zjjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:59:35.614178 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:35.614137    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556dcdc5-e48c-49d1-aa68-c1a25f938ccc-kube-api-access-bjhqt" pod "556dcdc5-e48c-49d1-aa68-c1a25f938ccc" (UID: "556dcdc5-e48c-49d1-aa68-c1a25f938ccc"). InnerVolumeSpecName "kube-api-access-bjhqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:59:35.712726 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:35.712636    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjhqt\" (UniqueName: \"kubernetes.io/projected/556dcdc5-e48c-49d1-aa68-c1a25f938ccc-kube-api-access-bjhqt\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:59:35.712726 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:35.712667    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zjjc\" (UniqueName: \"kubernetes.io/projected/3a464ae8-add5-47bb-8716-cd3ae6805e4a-kube-api-access-2zjjc\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:59:36.204182 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:36.204142    4294 generic.go:356] "Generic (PLEG): container finished" podID="556dcdc5-e48c-49d1-aa68-c1a25f938ccc" containerID="01dab2dd50cc331ae75d166da9c6b8344f61fa65e9b0ee069da2f660a6b6ca8f" exitCode=0
Nov 02 22:59:36.204182 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:36.204184    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-zm68f"
Nov 02 22:59:36.204692 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:36.204240    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-zm68f" event={"ID":"556dcdc5-e48c-49d1-aa68-c1a25f938ccc","Type":"ContainerDied","Data":"01dab2dd50cc331ae75d166da9c6b8344f61fa65e9b0ee069da2f660a6b6ca8f"}
Nov 02 22:59:36.204692 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:36.204280    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-zm68f" event={"ID":"556dcdc5-e48c-49d1-aa68-c1a25f938ccc","Type":"ContainerDied","Data":"3e7d74e374651e5bac9d528fddcde419d705471cac3dc11fb153451cb901cb56"}
Nov 02 22:59:36.204692 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:36.204304    4294 scope.go:122] "RemoveContainer" containerID="01dab2dd50cc331ae75d166da9c6b8344f61fa65e9b0ee069da2f660a6b6ca8f"
Nov 02 22:59:36.205531 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:36.205507    4294 generic.go:356] "Generic (PLEG): container finished" podID="3a464ae8-add5-47bb-8716-cd3ae6805e4a" containerID="b56a7635a57a04bb5784d51dda219c7890879aee47172f615973b4f5c67b3196" exitCode=0
Nov 02 22:59:36.205651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:36.205549    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69cf67b97b-t6vd6"
Nov 02 22:59:36.205651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:36.205591    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69cf67b97b-t6vd6" event={"ID":"3a464ae8-add5-47bb-8716-cd3ae6805e4a","Type":"ContainerDied","Data":"b56a7635a57a04bb5784d51dda219c7890879aee47172f615973b4f5c67b3196"}
Nov 02 22:59:36.205651 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:36.205631    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69cf67b97b-t6vd6" event={"ID":"3a464ae8-add5-47bb-8716-cd3ae6805e4a","Type":"ContainerDied","Data":"3c1c5e0977574301b02320c2379e60e7f6a8fafdade039a6e4bba32ff77a61c2"}
Nov 02 22:59:36.220037 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:36.219995    4294 scope.go:122] "RemoveContainer" containerID="01dab2dd50cc331ae75d166da9c6b8344f61fa65e9b0ee069da2f660a6b6ca8f"
Nov 02 22:59:36.220448 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:59:36.220419    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01dab2dd50cc331ae75d166da9c6b8344f61fa65e9b0ee069da2f660a6b6ca8f\": container with ID starting with 01dab2dd50cc331ae75d166da9c6b8344f61fa65e9b0ee069da2f660a6b6ca8f not found: ID does not exist" containerID="01dab2dd50cc331ae75d166da9c6b8344f61fa65e9b0ee069da2f660a6b6ca8f"
Nov 02 22:59:36.220533 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:36.220457    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01dab2dd50cc331ae75d166da9c6b8344f61fa65e9b0ee069da2f660a6b6ca8f"} err="failed to get container status \"01dab2dd50cc331ae75d166da9c6b8344f61fa65e9b0ee069da2f660a6b6ca8f\": rpc error: code = NotFound desc = could not find container \"01dab2dd50cc331ae75d166da9c6b8344f61fa65e9b0ee069da2f660a6b6ca8f\": container with ID starting with 01dab2dd50cc331ae75d166da9c6b8344f61fa65e9b0ee069da2f660a6b6ca8f not found: ID does not exist"
Nov 02 22:59:36.220533 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:36.220476    4294 scope.go:122] "RemoveContainer" containerID="b56a7635a57a04bb5784d51dda219c7890879aee47172f615973b4f5c67b3196"
Nov 02 22:59:36.228022 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:36.228003    4294 scope.go:122] "RemoveContainer" containerID="b56a7635a57a04bb5784d51dda219c7890879aee47172f615973b4f5c67b3196"
Nov 02 22:59:36.228293 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:59:36.228272    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56a7635a57a04bb5784d51dda219c7890879aee47172f615973b4f5c67b3196\": container with ID starting with b56a7635a57a04bb5784d51dda219c7890879aee47172f615973b4f5c67b3196 not found: ID does not exist" containerID="b56a7635a57a04bb5784d51dda219c7890879aee47172f615973b4f5c67b3196"
Nov 02 22:59:36.228335 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:36.228301    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56a7635a57a04bb5784d51dda219c7890879aee47172f615973b4f5c67b3196"} err="failed to get container status \"b56a7635a57a04bb5784d51dda219c7890879aee47172f615973b4f5c67b3196\": rpc error: code = NotFound desc = could not find container \"b56a7635a57a04bb5784d51dda219c7890879aee47172f615973b4f5c67b3196\": container with ID starting with b56a7635a57a04bb5784d51dda219c7890879aee47172f615973b4f5c67b3196 not found: ID does not exist"
Nov 02 22:59:36.360417 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:36.360380    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-69cf67b97b-t6vd6"]
Nov 02 22:59:36.364609 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:36.364579    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-69cf67b97b-t6vd6"]
Nov 02 22:59:36.468171 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:36.468081    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zm68f"]
Nov 02 22:59:36.471747 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:36.471718    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-zm68f"]
Nov 02 22:59:36.545317 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:36.545280    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a464ae8-add5-47bb-8716-cd3ae6805e4a" path="/var/lib/kubelet/pods/3a464ae8-add5-47bb-8716-cd3ae6805e4a/volumes"
Nov 02 22:59:36.545666 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:36.545646    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="556dcdc5-e48c-49d1-aa68-c1a25f938ccc" path="/var/lib/kubelet/pods/556dcdc5-e48c-49d1-aa68-c1a25f938ccc/volumes"
Nov 02 22:59:37.185161 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:37.185126    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"
Nov 02 22:59:37.343379 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:37.343345    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"]
Nov 02 22:59:37.343853 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:37.343595    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z" podUID="47339cf5-575d-4acf-b0f2-d4f64fe6ca3f" containerName="maas-api" containerID="cri-o://db0c99bea4790b55009af4ee228775731009ec7c03f61355526701d06b5ad56d" gracePeriod=30
Nov 02 22:59:37.588500 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:37.588475    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"
Nov 02 22:59:37.727794 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:37.727709    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-maas-api-tls\") pod \"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f\" (UID: \"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f\") "
Nov 02 22:59:37.727794 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:37.727750    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-kube-api-access-hc8ws\" (UniqueName: \"kubernetes.io/projected/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-kube-api-access-hc8ws\") pod \"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f\" (UID: \"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f\") "
Nov 02 22:59:37.727794 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:37.727784    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-openshift-service-ca\") pod \"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f\" (UID: \"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f\") "
Nov 02 22:59:37.728054 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:37.727872    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-metrics-certs\") pod \"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f\" (UID: \"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f\") "
Nov 02 22:59:37.728256 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:37.728231    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-openshift-service-ca" pod "47339cf5-575d-4acf-b0f2-d4f64fe6ca3f" (UID: "47339cf5-575d-4acf-b0f2-d4f64fe6ca3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 22:59:37.729998 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:37.729958    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-kube-api-access-hc8ws" pod "47339cf5-575d-4acf-b0f2-d4f64fe6ca3f" (UID: "47339cf5-575d-4acf-b0f2-d4f64fe6ca3f"). InnerVolumeSpecName "kube-api-access-hc8ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 22:59:37.730113 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:37.730015    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-metrics-certs" pod "47339cf5-575d-4acf-b0f2-d4f64fe6ca3f" (UID: "47339cf5-575d-4acf-b0f2-d4f64fe6ca3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:59:37.730113 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:37.730034    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-maas-api-tls" pod "47339cf5-575d-4acf-b0f2-d4f64fe6ca3f" (UID: "47339cf5-575d-4acf-b0f2-d4f64fe6ca3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 22:59:37.828569 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:37.828518    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:59:37.828569 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:37.828559    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:59:37.828569 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:37.828569    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc8ws\" (UniqueName: \"kubernetes.io/projected/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-kube-api-access-hc8ws\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:59:37.828569 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:37.828580    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 22:59:38.218181 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:38.218140    4294 generic.go:356] "Generic (PLEG): container finished" podID="47339cf5-575d-4acf-b0f2-d4f64fe6ca3f" containerID="db0c99bea4790b55009af4ee228775731009ec7c03f61355526701d06b5ad56d" exitCode=0
Nov 02 22:59:38.218382 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:38.218185    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"
Nov 02 22:59:38.218382 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:38.218230    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z" event={"ID":"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f","Type":"ContainerDied","Data":"db0c99bea4790b55009af4ee228775731009ec7c03f61355526701d06b5ad56d"}
Nov 02 22:59:38.218382 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:38.218289    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z" event={"ID":"47339cf5-575d-4acf-b0f2-d4f64fe6ca3f","Type":"ContainerDied","Data":"7f235fdb6d65676b86ee087b5f2b77588cbadcf117876fd274cffb53f6840e82"}
Nov 02 22:59:38.218382 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:38.218313    4294 scope.go:122] "RemoveContainer" containerID="db0c99bea4790b55009af4ee228775731009ec7c03f61355526701d06b5ad56d"
Nov 02 22:59:38.226946 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:38.226922    4294 scope.go:122] "RemoveContainer" containerID="db0c99bea4790b55009af4ee228775731009ec7c03f61355526701d06b5ad56d"
Nov 02 22:59:38.227305 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:59:38.227287    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0c99bea4790b55009af4ee228775731009ec7c03f61355526701d06b5ad56d\": container with ID starting with db0c99bea4790b55009af4ee228775731009ec7c03f61355526701d06b5ad56d not found: ID does not exist" containerID="db0c99bea4790b55009af4ee228775731009ec7c03f61355526701d06b5ad56d"
Nov 02 22:59:38.227360 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:38.227313    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0c99bea4790b55009af4ee228775731009ec7c03f61355526701d06b5ad56d"} err="failed to get container status \"db0c99bea4790b55009af4ee228775731009ec7c03f61355526701d06b5ad56d\": rpc error: code = NotFound desc = could not find container \"db0c99bea4790b55009af4ee228775731009ec7c03f61355526701d06b5ad56d\": container with ID starting with db0c99bea4790b55009af4ee228775731009ec7c03f61355526701d06b5ad56d not found: ID does not exist"
Nov 02 22:59:38.374598 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:38.374561    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"]
Nov 02 22:59:38.422237 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:38.422199    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ccf48f65-pbk4z"]
Nov 02 22:59:38.545161 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:38.545132    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47339cf5-575d-4acf-b0f2-d4f64fe6ca3f" path="/var/lib/kubelet/pods/47339cf5-575d-4acf-b0f2-d4f64fe6ca3f/volumes"
Nov 02 22:59:41.748301 ip-10-0-139-234 kubenswrapper[4294]: E1102 22:59:41.748263    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 22:59:48.748617 ip-10-0-139-234 kubenswrapper[4294]: I1102 22:59:48.748580    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:00:11.748177 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:00:11.748136    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:00:23.397016 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:23.396962    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:00:38.091904 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:38.091870    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:00:41.748130 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:00:41.748091    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:00:51.126234 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.126196    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-f68kh"]
Nov 02 23:00:51.126873 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.126643    4294 cpu_manager.go:414] "containerMap: removing container" podUID="556dcdc5-e48c-49d1-aa68-c1a25f938ccc" containerName="authorino"
Nov 02 23:00:51.126873 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.126662    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="556dcdc5-e48c-49d1-aa68-c1a25f938ccc" containerName="authorino"
Nov 02 23:00:51.126873 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.126696    4294 cpu_manager.go:414] "containerMap: removing container" podUID="47339cf5-575d-4acf-b0f2-d4f64fe6ca3f" containerName="maas-api"
Nov 02 23:00:51.126873 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.126703    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47339cf5-575d-4acf-b0f2-d4f64fe6ca3f" containerName="maas-api"
Nov 02 23:00:51.126873 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.126712    4294 cpu_manager.go:414] "containerMap: removing container" podUID="3a464ae8-add5-47bb-8716-cd3ae6805e4a" containerName="authorino"
Nov 02 23:00:51.126873 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.126719    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a464ae8-add5-47bb-8716-cd3ae6805e4a" containerName="authorino"
Nov 02 23:00:51.126873 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.126808    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="47339cf5-575d-4acf-b0f2-d4f64fe6ca3f" containerName="maas-api"
Nov 02 23:00:51.126873 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.126817    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a464ae8-add5-47bb-8716-cd3ae6805e4a" containerName="authorino"
Nov 02 23:00:51.126873 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.126831    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="556dcdc5-e48c-49d1-aa68-c1a25f938ccc" containerName="authorino"
Nov 02 23:00:51.128957 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.128937    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-f68kh"
Nov 02 23:00:51.162216 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.162184    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:00:51.216165 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.216131    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d75254ad-c7e5-4a59-8a7d-288a1024f1db-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-f68kh\" (UID: \"d75254ad-c7e5-4a59-8a7d-288a1024f1db\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-f68kh"
Nov 02 23:00:51.216357 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.216193    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tznpf\" (UniqueName: \"kubernetes.io/projected/d75254ad-c7e5-4a59-8a7d-288a1024f1db-kube-api-access-tznpf\") pod \"ai-gateway-controller-5d8dddb676-f68kh\" (UID: \"d75254ad-c7e5-4a59-8a7d-288a1024f1db\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-f68kh"
Nov 02 23:00:51.226732 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.226700    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 02 23:00:51.226732 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.226722    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 23:00:51.226732 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.226723    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-22k6h\""
Nov 02 23:00:51.228476 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.228454    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-f68kh"]
Nov 02 23:00:51.317692 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.317659    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tznpf\" (UniqueName: \"kubernetes.io/projected/d75254ad-c7e5-4a59-8a7d-288a1024f1db-kube-api-access-tznpf\") pod \"ai-gateway-controller-5d8dddb676-f68kh\" (UID: \"d75254ad-c7e5-4a59-8a7d-288a1024f1db\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-f68kh"
Nov 02 23:00:51.317865 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.317744    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d75254ad-c7e5-4a59-8a7d-288a1024f1db-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-f68kh\" (UID: \"d75254ad-c7e5-4a59-8a7d-288a1024f1db\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-f68kh"
Nov 02 23:00:51.320047 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.320023    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d75254ad-c7e5-4a59-8a7d-288a1024f1db-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-f68kh\" (UID: \"d75254ad-c7e5-4a59-8a7d-288a1024f1db\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-f68kh"
Nov 02 23:00:51.357193 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.357162    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:00:51.372451 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.372420    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tznpf\" (UniqueName: \"kubernetes.io/projected/d75254ad-c7e5-4a59-8a7d-288a1024f1db-kube-api-access-tznpf\") pod \"ai-gateway-controller-5d8dddb676-f68kh\" (UID: \"d75254ad-c7e5-4a59-8a7d-288a1024f1db\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-f68kh"
Nov 02 23:00:51.439777 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.439679    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-f68kh"
Nov 02 23:00:51.803584 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:51.803561    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-f68kh"]
Nov 02 23:00:51.805275 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:00:51.805240    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75254ad_c7e5_4a59_8a7d_288a1024f1db.slice/crio-bb7564eba8c7a9ce7e64d57a4d54fe4c298c3bb26f9547f514a1a81f6c7ed777 WatchSource:0}: Error finding container bb7564eba8c7a9ce7e64d57a4d54fe4c298c3bb26f9547f514a1a81f6c7ed777: Status 404 returned error can't find the container with id bb7564eba8c7a9ce7e64d57a4d54fe4c298c3bb26f9547f514a1a81f6c7ed777
Nov 02 23:00:52.512839 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:52.512803    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-f68kh" event={"ID":"d75254ad-c7e5-4a59-8a7d-288a1024f1db","Type":"ContainerStarted","Data":"bb7564eba8c7a9ce7e64d57a4d54fe4c298c3bb26f9547f514a1a81f6c7ed777"}
Nov 02 23:00:54.522716 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:54.522679    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-f68kh" event={"ID":"d75254ad-c7e5-4a59-8a7d-288a1024f1db","Type":"ContainerStarted","Data":"40c49ae1f1d58aba118ccf501a3f4d3113adfdad68d817e0f9ac8329f94e5e55"}
Nov 02 23:00:54.523156 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:54.522781    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-f68kh"
Nov 02 23:00:54.634742 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:54.634680    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-f68kh" podStartSLOduration=1.542681449 podStartE2EDuration="3.634664193s" totalImagesPullingTime="2.091982744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:00:54.633722636 +0000 UTC m=+494.202077423" watchObservedRunningTime="2026-11-02 23:00:54.634664193 +0000 UTC m=+494.203018980"
Nov 02 23:00:56.741954 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:56.741917    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:00:56.903185 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:56.903152    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:00:57.105936 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.105857    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l"]
Nov 02 23:00:57.108020 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.108003    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l"
Nov 02 23:00:57.208445 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.208405    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-b8n25\""
Nov 02 23:00:57.208445 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.208421    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 02 23:00:57.209933 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.209913    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l"]
Nov 02 23:00:57.257829 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.257788    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rnnmk"]
Nov 02 23:00:57.260191 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.260173    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-rnnmk"
Nov 02 23:00:57.271681 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.271658    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939ab75a-b60a-408d-8661-1f4e17c4b809-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-c7k9l\" (UID: \"939ab75a-b60a-408d-8661-1f4e17c4b809\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l"
Nov 02 23:00:57.271786 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.271728    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9x4j\" (UniqueName: \"kubernetes.io/projected/939ab75a-b60a-408d-8661-1f4e17c4b809-kube-api-access-v9x4j\") pod \"payload-pre-processing-7b98bb678c-c7k9l\" (UID: \"939ab75a-b60a-408d-8661-1f4e17c4b809\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l"
Nov 02 23:00:57.307101 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.307068    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rnnmk"]
Nov 02 23:00:57.373234 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.373145    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr42k\" (UniqueName: \"kubernetes.io/projected/cab95f3e-f51d-45b6-a218-e0c272a09aac-kube-api-access-sr42k\") pod \"payload-processing-55fbbbf9dc-rnnmk\" (UID: \"cab95f3e-f51d-45b6-a218-e0c272a09aac\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rnnmk"
Nov 02 23:00:57.373234 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.373192    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab95f3e-f51d-45b6-a218-e0c272a09aac-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rnnmk\" (UID: \"cab95f3e-f51d-45b6-a218-e0c272a09aac\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rnnmk"
Nov 02 23:00:57.373425 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.373299    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9x4j\" (UniqueName: \"kubernetes.io/projected/939ab75a-b60a-408d-8661-1f4e17c4b809-kube-api-access-v9x4j\") pod \"payload-pre-processing-7b98bb678c-c7k9l\" (UID: \"939ab75a-b60a-408d-8661-1f4e17c4b809\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l"
Nov 02 23:00:57.373425 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.373347    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939ab75a-b60a-408d-8661-1f4e17c4b809-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-c7k9l\" (UID: \"939ab75a-b60a-408d-8661-1f4e17c4b809\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l"
Nov 02 23:00:57.374015 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.373970    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939ab75a-b60a-408d-8661-1f4e17c4b809-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-c7k9l\" (UID: \"939ab75a-b60a-408d-8661-1f4e17c4b809\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l"
Nov 02 23:00:57.427627 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.427590    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9x4j\" (UniqueName: \"kubernetes.io/projected/939ab75a-b60a-408d-8661-1f4e17c4b809-kube-api-access-v9x4j\") pod \"payload-pre-processing-7b98bb678c-c7k9l\" (UID: \"939ab75a-b60a-408d-8661-1f4e17c4b809\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l"
Nov 02 23:00:57.474428 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.474389    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab95f3e-f51d-45b6-a218-e0c272a09aac-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rnnmk\" (UID: \"cab95f3e-f51d-45b6-a218-e0c272a09aac\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rnnmk"
Nov 02 23:00:57.474604 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.474495    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr42k\" (UniqueName: \"kubernetes.io/projected/cab95f3e-f51d-45b6-a218-e0c272a09aac-kube-api-access-sr42k\") pod \"payload-processing-55fbbbf9dc-rnnmk\" (UID: \"cab95f3e-f51d-45b6-a218-e0c272a09aac\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rnnmk"
Nov 02 23:00:57.475132 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.475111    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab95f3e-f51d-45b6-a218-e0c272a09aac-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rnnmk\" (UID: \"cab95f3e-f51d-45b6-a218-e0c272a09aac\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rnnmk"
Nov 02 23:00:57.528635 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.528594    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr42k\" (UniqueName: \"kubernetes.io/projected/cab95f3e-f51d-45b6-a218-e0c272a09aac-kube-api-access-sr42k\") pod \"payload-processing-55fbbbf9dc-rnnmk\" (UID: \"cab95f3e-f51d-45b6-a218-e0c272a09aac\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rnnmk"
Nov 02 23:00:57.571680 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.571640    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-rnnmk"
Nov 02 23:00:57.719082 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.719051    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l"
Nov 02 23:00:57.935237 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:57.935212    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rnnmk"]
Nov 02 23:00:57.935926 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:00:57.935898    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab95f3e_f51d_45b6_a218_e0c272a09aac.slice/crio-3b78e03cb0c6b0a750b103bd8038e681b6106f6cf53342be92f55b9dc82fc7a9 WatchSource:0}: Error finding container 3b78e03cb0c6b0a750b103bd8038e681b6106f6cf53342be92f55b9dc82fc7a9: Status 404 returned error can't find the container with id 3b78e03cb0c6b0a750b103bd8038e681b6106f6cf53342be92f55b9dc82fc7a9
Nov 02 23:00:58.078696 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:58.078670    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l"]
Nov 02 23:00:58.080866 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:00:58.080833    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod939ab75a_b60a_408d_8661_1f4e17c4b809.slice/crio-c7fa4711aa05e1571aa43339158d221761cd8585cbfdf1ccdaec2f608eb53670 WatchSource:0}: Error finding container c7fa4711aa05e1571aa43339158d221761cd8585cbfdf1ccdaec2f608eb53670: Status 404 returned error can't find the container with id c7fa4711aa05e1571aa43339158d221761cd8585cbfdf1ccdaec2f608eb53670
Nov 02 23:00:58.545159 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:58.545122    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-rnnmk" event={"ID":"cab95f3e-f51d-45b6-a218-e0c272a09aac","Type":"ContainerStarted","Data":"3b78e03cb0c6b0a750b103bd8038e681b6106f6cf53342be92f55b9dc82fc7a9"}
Nov 02 23:00:58.545309 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:00:58.545168    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l" event={"ID":"939ab75a-b60a-408d-8661-1f4e17c4b809","Type":"ContainerStarted","Data":"c7fa4711aa05e1571aa43339158d221761cd8585cbfdf1ccdaec2f608eb53670"}
Nov 02 23:01:01.560449 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:01.560411    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-rnnmk" event={"ID":"cab95f3e-f51d-45b6-a218-e0c272a09aac","Type":"ContainerStarted","Data":"eb3d73a0b4f614accf15ec9c40c2d1c88c796d2aeb56082a5e029dde8eddc3f0"}
Nov 02 23:01:01.560932 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:01.560576    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-rnnmk"
Nov 02 23:01:01.561901 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:01.561878    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l" event={"ID":"939ab75a-b60a-408d-8661-1f4e17c4b809","Type":"ContainerStarted","Data":"b5a5b590b70fbd9d7e790edccc8e3da7366e7d90bd6d0cb21642a32fd806b653"}
Nov 02 23:01:01.562032 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:01.561928    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l"
Nov 02 23:01:01.645970 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:01.645933    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:01:01.671905 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:01.671856    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-rnnmk" podStartSLOduration=1.6110516860000001 podStartE2EDuration="4.671843584s" totalImagesPullingTime="3.060791898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:01.67135044 +0000 UTC m=+501.239705228" watchObservedRunningTime="2026-11-02 23:01:01.671843584 +0000 UTC m=+501.240198370"
Nov 02 23:01:01.781222 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:01.781161    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l" podStartSLOduration=1.957401473 podStartE2EDuration="4.781147323s" totalImagesPullingTime="2.82374585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:01.779968304 +0000 UTC m=+501.348323094" watchObservedRunningTime="2026-11-02 23:01:01.781147323 +0000 UTC m=+501.349502110"
Nov 02 23:01:05.529431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:05.529401    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-f68kh"
Nov 02 23:01:05.873570 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:05.873483    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-669b4bd578-5cj8x"]
Nov 02 23:01:05.875973 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:05.875955    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-669b4bd578-5cj8x"
Nov 02 23:01:05.930686 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:05.930656    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-669b4bd578-5cj8x"]
Nov 02 23:01:05.949243 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:05.949196    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4p46\" (UniqueName: \"kubernetes.io/projected/ac0193db-08d3-4abb-9a82-c2199ba08ccf-kube-api-access-s4p46\") pod \"ai-gateway-controller-669b4bd578-5cj8x\" (UID: \"ac0193db-08d3-4abb-9a82-c2199ba08ccf\") " pod="opendatahub/ai-gateway-controller-669b4bd578-5cj8x"
Nov 02 23:01:05.949405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:05.949336    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac0193db-08d3-4abb-9a82-c2199ba08ccf-metrics-certs\") pod \"ai-gateway-controller-669b4bd578-5cj8x\" (UID: \"ac0193db-08d3-4abb-9a82-c2199ba08ccf\") " pod="opendatahub/ai-gateway-controller-669b4bd578-5cj8x"
Nov 02 23:01:06.050444 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:06.050403    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac0193db-08d3-4abb-9a82-c2199ba08ccf-metrics-certs\") pod \"ai-gateway-controller-669b4bd578-5cj8x\" (UID: \"ac0193db-08d3-4abb-9a82-c2199ba08ccf\") " pod="opendatahub/ai-gateway-controller-669b4bd578-5cj8x"
Nov 02 23:01:06.050649 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:06.050468    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4p46\" (UniqueName: \"kubernetes.io/projected/ac0193db-08d3-4abb-9a82-c2199ba08ccf-kube-api-access-s4p46\") pod \"ai-gateway-controller-669b4bd578-5cj8x\" (UID: \"ac0193db-08d3-4abb-9a82-c2199ba08ccf\") " pod="opendatahub/ai-gateway-controller-669b4bd578-5cj8x"
Nov 02 23:01:06.052955 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:06.052928    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac0193db-08d3-4abb-9a82-c2199ba08ccf-metrics-certs\") pod \"ai-gateway-controller-669b4bd578-5cj8x\" (UID: \"ac0193db-08d3-4abb-9a82-c2199ba08ccf\") " pod="opendatahub/ai-gateway-controller-669b4bd578-5cj8x"
Nov 02 23:01:06.106532 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:06.106498    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4p46\" (UniqueName: \"kubernetes.io/projected/ac0193db-08d3-4abb-9a82-c2199ba08ccf-kube-api-access-s4p46\") pod \"ai-gateway-controller-669b4bd578-5cj8x\" (UID: \"ac0193db-08d3-4abb-9a82-c2199ba08ccf\") " pod="opendatahub/ai-gateway-controller-669b4bd578-5cj8x"
Nov 02 23:01:06.187467 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:06.187380    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-669b4bd578-5cj8x"
Nov 02 23:01:06.548729 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:06.548700    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-669b4bd578-5cj8x"]
Nov 02 23:01:06.549881 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:01:06.549854    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0193db_08d3_4abb_9a82_c2199ba08ccf.slice/crio-a1d4cf3d06c46338122eaa7c8e275decb16eb44a921008672a3561f6f3208456 WatchSource:0}: Error finding container a1d4cf3d06c46338122eaa7c8e275decb16eb44a921008672a3561f6f3208456: Status 404 returned error can't find the container with id a1d4cf3d06c46338122eaa7c8e275decb16eb44a921008672a3561f6f3208456
Nov 02 23:01:06.583581 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:06.583545    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-669b4bd578-5cj8x" event={"ID":"ac0193db-08d3-4abb-9a82-c2199ba08ccf","Type":"ContainerStarted","Data":"a1d4cf3d06c46338122eaa7c8e275decb16eb44a921008672a3561f6f3208456"}
Nov 02 23:01:07.589166 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:07.589126    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-669b4bd578-5cj8x" event={"ID":"ac0193db-08d3-4abb-9a82-c2199ba08ccf","Type":"ContainerStarted","Data":"99dc384be10db979917cda06ecd210a8a2cade6464dbad7ce182e545a386b498"}
Nov 02 23:01:07.589538 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:07.589207    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-669b4bd578-5cj8x"
Nov 02 23:01:07.701293 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:07.701172    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-669b4bd578-5cj8x" podStartSLOduration=1.835797682 podStartE2EDuration="2.701158857s" totalImagesPullingTime="865.361175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:07.699518836 +0000 UTC m=+507.267873624" watchObservedRunningTime="2026-11-02 23:01:07.701158857 +0000 UTC m=+507.269513661"
Nov 02 23:01:11.748429 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:01:11.748390    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:01:12.569308 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:12.569280    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-rnnmk"
Nov 02 23:01:12.569489 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:12.569343    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l"
Nov 02 23:01:12.675627 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:12.675596    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:01:18.596217 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:18.596182    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-669b4bd578-5cj8x"
Nov 02 23:01:18.729391 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:18.729353    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-f68kh"]
Nov 02 23:01:18.729587 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:18.729568    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-f68kh" podUID="d75254ad-c7e5-4a59-8a7d-288a1024f1db" containerName="manager" containerID="cri-o://40c49ae1f1d58aba118ccf501a3f4d3113adfdad68d817e0f9ac8329f94e5e55" gracePeriod=30
Nov 02 23:01:18.965885 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:18.965861    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-f68kh"
Nov 02 23:01:19.060000 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:19.059938    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d75254ad-c7e5-4a59-8a7d-288a1024f1db-kube-api-access-tznpf\" (UniqueName: \"kubernetes.io/projected/d75254ad-c7e5-4a59-8a7d-288a1024f1db-kube-api-access-tznpf\") pod \"d75254ad-c7e5-4a59-8a7d-288a1024f1db\" (UID: \"d75254ad-c7e5-4a59-8a7d-288a1024f1db\") "
Nov 02 23:01:19.060175 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:19.060018    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d75254ad-c7e5-4a59-8a7d-288a1024f1db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d75254ad-c7e5-4a59-8a7d-288a1024f1db-metrics-certs\") pod \"d75254ad-c7e5-4a59-8a7d-288a1024f1db\" (UID: \"d75254ad-c7e5-4a59-8a7d-288a1024f1db\") "
Nov 02 23:01:19.062218 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:19.062189    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75254ad-c7e5-4a59-8a7d-288a1024f1db-kube-api-access-tznpf" pod "d75254ad-c7e5-4a59-8a7d-288a1024f1db" (UID: "d75254ad-c7e5-4a59-8a7d-288a1024f1db"). InnerVolumeSpecName "kube-api-access-tznpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:01:19.062335 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:19.062314    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d75254ad-c7e5-4a59-8a7d-288a1024f1db-metrics-certs" pod "d75254ad-c7e5-4a59-8a7d-288a1024f1db" (UID: "d75254ad-c7e5-4a59-8a7d-288a1024f1db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:01:19.160936 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:19.160845    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tznpf\" (UniqueName: \"kubernetes.io/projected/d75254ad-c7e5-4a59-8a7d-288a1024f1db-kube-api-access-tznpf\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:01:19.160936 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:19.160882    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d75254ad-c7e5-4a59-8a7d-288a1024f1db-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:01:19.638405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:19.638370    4294 generic.go:356] "Generic (PLEG): container finished" podID="d75254ad-c7e5-4a59-8a7d-288a1024f1db" containerID="40c49ae1f1d58aba118ccf501a3f4d3113adfdad68d817e0f9ac8329f94e5e55" exitCode=0
Nov 02 23:01:19.638865 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:19.638415    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-f68kh"
Nov 02 23:01:19.638865 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:19.638432    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-f68kh" event={"ID":"d75254ad-c7e5-4a59-8a7d-288a1024f1db","Type":"ContainerDied","Data":"40c49ae1f1d58aba118ccf501a3f4d3113adfdad68d817e0f9ac8329f94e5e55"}
Nov 02 23:01:19.638865 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:19.638486    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-f68kh" event={"ID":"d75254ad-c7e5-4a59-8a7d-288a1024f1db","Type":"ContainerDied","Data":"bb7564eba8c7a9ce7e64d57a4d54fe4c298c3bb26f9547f514a1a81f6c7ed777"}
Nov 02 23:01:19.638865 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:19.638549    4294 scope.go:122] "RemoveContainer" containerID="40c49ae1f1d58aba118ccf501a3f4d3113adfdad68d817e0f9ac8329f94e5e55"
Nov 02 23:01:19.646998 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:19.646963    4294 scope.go:122] "RemoveContainer" containerID="40c49ae1f1d58aba118ccf501a3f4d3113adfdad68d817e0f9ac8329f94e5e55"
Nov 02 23:01:19.647328 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:01:19.647309    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c49ae1f1d58aba118ccf501a3f4d3113adfdad68d817e0f9ac8329f94e5e55\": container with ID starting with 40c49ae1f1d58aba118ccf501a3f4d3113adfdad68d817e0f9ac8329f94e5e55 not found: ID does not exist" containerID="40c49ae1f1d58aba118ccf501a3f4d3113adfdad68d817e0f9ac8329f94e5e55"
Nov 02 23:01:19.647373 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:19.647339    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c49ae1f1d58aba118ccf501a3f4d3113adfdad68d817e0f9ac8329f94e5e55"} err="failed to get container status \"40c49ae1f1d58aba118ccf501a3f4d3113adfdad68d817e0f9ac8329f94e5e55\": rpc error: code = NotFound desc = could not find container \"40c49ae1f1d58aba118ccf501a3f4d3113adfdad68d817e0f9ac8329f94e5e55\": container with ID starting with 40c49ae1f1d58aba118ccf501a3f4d3113adfdad68d817e0f9ac8329f94e5e55 not found: ID does not exist"
Nov 02 23:01:19.794903 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:19.794870    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-f68kh"]
Nov 02 23:01:19.842520 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:19.842480    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-f68kh"]
Nov 02 23:01:20.264139 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:20.264102    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:01:20.545571 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:20.545477    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d75254ad-c7e5-4a59-8a7d-288a1024f1db" path="/var/lib/kubelet/pods/d75254ad-c7e5-4a59-8a7d-288a1024f1db/volumes"
Nov 02 23:01:20.738584 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:20.738549    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:01:21.882097 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:21.882047    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:01:22.062898 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:22.062859    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:01:23.728286 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:23.728252    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:01:28.645882 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.645851    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"]
Nov 02 23:01:28.648646 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.646757    4294 cpu_manager.go:414] "containerMap: removing container" podUID="d75254ad-c7e5-4a59-8a7d-288a1024f1db" containerName="manager"
Nov 02 23:01:28.648646 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.646787    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d75254ad-c7e5-4a59-8a7d-288a1024f1db" containerName="manager"
Nov 02 23:01:28.648646 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.647044    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d75254ad-c7e5-4a59-8a7d-288a1024f1db" containerName="manager"
Nov 02 23:01:28.652272 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.652233    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:28.702368 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.702336    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"]
Nov 02 23:01:28.729324 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.729294    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 23:01:28.748422 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.748381    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f2848e7-5412-41cf-a47f-4d33a355e417-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k\" (UID: \"2f2848e7-5412-41cf-a47f-4d33a355e417\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:28.748422 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.748403    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 23:01:28.748422 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.748424    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq24t\" (UniqueName: \"kubernetes.io/projected/2f2848e7-5412-41cf-a47f-4d33a355e417-kube-api-access-kq24t\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k\" (UID: \"2f2848e7-5412-41cf-a47f-4d33a355e417\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:28.748692 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.748399    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cnbg7\""
Nov 02 23:01:28.748692 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.748514    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 02 23:01:28.748692 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.748528    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f2848e7-5412-41cf-a47f-4d33a355e417-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k\" (UID: \"2f2848e7-5412-41cf-a47f-4d33a355e417\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:28.748692 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.748561    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f2848e7-5412-41cf-a47f-4d33a355e417-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k\" (UID: \"2f2848e7-5412-41cf-a47f-4d33a355e417\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:28.748692 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.748638    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f2848e7-5412-41cf-a47f-4d33a355e417-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k\" (UID: \"2f2848e7-5412-41cf-a47f-4d33a355e417\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:28.748692 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.748667    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f2848e7-5412-41cf-a47f-4d33a355e417-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k\" (UID: \"2f2848e7-5412-41cf-a47f-4d33a355e417\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:28.791455 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.791422    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:01:28.849148 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.849114    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f2848e7-5412-41cf-a47f-4d33a355e417-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k\" (UID: \"2f2848e7-5412-41cf-a47f-4d33a355e417\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:28.849343 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.849154    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq24t\" (UniqueName: \"kubernetes.io/projected/2f2848e7-5412-41cf-a47f-4d33a355e417-kube-api-access-kq24t\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k\" (UID: \"2f2848e7-5412-41cf-a47f-4d33a355e417\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:28.849343 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.849202    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f2848e7-5412-41cf-a47f-4d33a355e417-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k\" (UID: \"2f2848e7-5412-41cf-a47f-4d33a355e417\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:28.849343 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.849225    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f2848e7-5412-41cf-a47f-4d33a355e417-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k\" (UID: \"2f2848e7-5412-41cf-a47f-4d33a355e417\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:28.849343 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.849241    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f2848e7-5412-41cf-a47f-4d33a355e417-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k\" (UID: \"2f2848e7-5412-41cf-a47f-4d33a355e417\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:28.849343 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.849266    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f2848e7-5412-41cf-a47f-4d33a355e417-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k\" (UID: \"2f2848e7-5412-41cf-a47f-4d33a355e417\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:28.849702 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.849678    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f2848e7-5412-41cf-a47f-4d33a355e417-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k\" (UID: \"2f2848e7-5412-41cf-a47f-4d33a355e417\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:28.849702 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.849696    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f2848e7-5412-41cf-a47f-4d33a355e417-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k\" (UID: \"2f2848e7-5412-41cf-a47f-4d33a355e417\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:28.849809 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.849755    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f2848e7-5412-41cf-a47f-4d33a355e417-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k\" (UID: \"2f2848e7-5412-41cf-a47f-4d33a355e417\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:28.851593 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.851570    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f2848e7-5412-41cf-a47f-4d33a355e417-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k\" (UID: \"2f2848e7-5412-41cf-a47f-4d33a355e417\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:28.851756 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.851739    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f2848e7-5412-41cf-a47f-4d33a355e417-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k\" (UID: \"2f2848e7-5412-41cf-a47f-4d33a355e417\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:28.903820 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.903735    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq24t\" (UniqueName: \"kubernetes.io/projected/2f2848e7-5412-41cf-a47f-4d33a355e417-kube-api-access-kq24t\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k\" (UID: \"2f2848e7-5412-41cf-a47f-4d33a355e417\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:28.965920 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:28.965881    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:29.329692 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:29.329667    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"]
Nov 02 23:01:29.331425 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:01:29.331393    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2848e7_5412_41cf_a47f_4d33a355e417.slice/crio-75fbfadc50032d0bb450464d283df9f05295f630aa7bfdcc852ecf07cc9ed09c WatchSource:0}: Error finding container 75fbfadc50032d0bb450464d283df9f05295f630aa7bfdcc852ecf07cc9ed09c: Status 404 returned error can't find the container with id 75fbfadc50032d0bb450464d283df9f05295f630aa7bfdcc852ecf07cc9ed09c
Nov 02 23:01:29.683298 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:29.683198    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k" event={"ID":"2f2848e7-5412-41cf-a47f-4d33a355e417","Type":"ContainerStarted","Data":"75fbfadc50032d0bb450464d283df9f05295f630aa7bfdcc852ecf07cc9ed09c"}
Nov 02 23:01:32.700887 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:32.700852    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k" event={"ID":"2f2848e7-5412-41cf-a47f-4d33a355e417","Type":"ContainerStarted","Data":"8bdf2f2c92251e2afba85defd591b1fec8662214d33834b12335ad7608c5598c"}
Nov 02 23:01:32.811912 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:32.811853    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k" podStartSLOduration=2.420815233 podStartE2EDuration="4.811834139s" totalImagesPullingTime="2.391018906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:32.809884198 +0000 UTC m=+532.378239028" watchObservedRunningTime="2026-11-02 23:01:32.811834139 +0000 UTC m=+532.380188927"
Nov 02 23:01:32.966769 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:32.966682    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:35.118924 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.118889    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"]
Nov 02 23:01:35.126410 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.126386    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:35.176773 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.176732    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 02 23:01:35.177356 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.177325    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"]
Nov 02 23:01:35.209201 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.209163    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a2abc73-1645-4b27-bb93-bbf3091cde02-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-twdqr\" (UID: \"6a2abc73-1645-4b27-bb93-bbf3091cde02\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:35.209392 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.209214    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a2abc73-1645-4b27-bb93-bbf3091cde02-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-twdqr\" (UID: \"6a2abc73-1645-4b27-bb93-bbf3091cde02\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:35.209392 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.209233    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2abc73-1645-4b27-bb93-bbf3091cde02-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-twdqr\" (UID: \"6a2abc73-1645-4b27-bb93-bbf3091cde02\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:35.209392 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.209328    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwnsj\" (UniqueName: \"kubernetes.io/projected/6a2abc73-1645-4b27-bb93-bbf3091cde02-kube-api-access-cwnsj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-twdqr\" (UID: \"6a2abc73-1645-4b27-bb93-bbf3091cde02\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:35.209392 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.209353    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a2abc73-1645-4b27-bb93-bbf3091cde02-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-twdqr\" (UID: \"6a2abc73-1645-4b27-bb93-bbf3091cde02\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:35.209392 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.209374    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a2abc73-1645-4b27-bb93-bbf3091cde02-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-twdqr\" (UID: \"6a2abc73-1645-4b27-bb93-bbf3091cde02\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:35.242631 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.242593    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:01:35.310786 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.310744    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a2abc73-1645-4b27-bb93-bbf3091cde02-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-twdqr\" (UID: \"6a2abc73-1645-4b27-bb93-bbf3091cde02\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:35.310919 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.310799    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a2abc73-1645-4b27-bb93-bbf3091cde02-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-twdqr\" (UID: \"6a2abc73-1645-4b27-bb93-bbf3091cde02\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:35.310919 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.310815    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2abc73-1645-4b27-bb93-bbf3091cde02-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-twdqr\" (UID: \"6a2abc73-1645-4b27-bb93-bbf3091cde02\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:35.310919 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.310835    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwnsj\" (UniqueName: \"kubernetes.io/projected/6a2abc73-1645-4b27-bb93-bbf3091cde02-kube-api-access-cwnsj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-twdqr\" (UID: \"6a2abc73-1645-4b27-bb93-bbf3091cde02\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:35.310919 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.310858    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a2abc73-1645-4b27-bb93-bbf3091cde02-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-twdqr\" (UID: \"6a2abc73-1645-4b27-bb93-bbf3091cde02\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:35.311150 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.311047    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a2abc73-1645-4b27-bb93-bbf3091cde02-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-twdqr\" (UID: \"6a2abc73-1645-4b27-bb93-bbf3091cde02\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:35.311375 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.311351    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a2abc73-1645-4b27-bb93-bbf3091cde02-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-twdqr\" (UID: \"6a2abc73-1645-4b27-bb93-bbf3091cde02\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:35.311375 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.311350    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a2abc73-1645-4b27-bb93-bbf3091cde02-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-twdqr\" (UID: \"6a2abc73-1645-4b27-bb93-bbf3091cde02\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:35.311526 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.311398    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a2abc73-1645-4b27-bb93-bbf3091cde02-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-twdqr\" (UID: \"6a2abc73-1645-4b27-bb93-bbf3091cde02\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:35.313152 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.313126    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a2abc73-1645-4b27-bb93-bbf3091cde02-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-twdqr\" (UID: \"6a2abc73-1645-4b27-bb93-bbf3091cde02\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:35.313384 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.313368    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a2abc73-1645-4b27-bb93-bbf3091cde02-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-twdqr\" (UID: \"6a2abc73-1645-4b27-bb93-bbf3091cde02\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:35.370877 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.370798    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwnsj\" (UniqueName: \"kubernetes.io/projected/6a2abc73-1645-4b27-bb93-bbf3091cde02-kube-api-access-cwnsj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-twdqr\" (UID: \"6a2abc73-1645-4b27-bb93-bbf3091cde02\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:35.438860 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.438822    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:35.803948 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:35.803923    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"]
Nov 02 23:01:35.805287 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:01:35.805258    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2abc73_1645_4b27_bb93_bbf3091cde02.slice/crio-a9ae603871f14d6bd3e182289d7f77093cd1103f3225f3128346561cbea29fb8 WatchSource:0}: Error finding container a9ae603871f14d6bd3e182289d7f77093cd1103f3225f3128346561cbea29fb8: Status 404 returned error can't find the container with id a9ae603871f14d6bd3e182289d7f77093cd1103f3225f3128346561cbea29fb8
Nov 02 23:01:36.720584 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:36.720492    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr" event={"ID":"6a2abc73-1645-4b27-bb93-bbf3091cde02","Type":"ContainerStarted","Data":"a4e5449831dddc208cfe5eb12263b4900212821fcdba79915c598f710c4a9b5c"}
Nov 02 23:01:36.720584 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:36.720542    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr" event={"ID":"6a2abc73-1645-4b27-bb93-bbf3091cde02","Type":"ContainerStarted","Data":"a9ae603871f14d6bd3e182289d7f77093cd1103f3225f3128346561cbea29fb8"}
Nov 02 23:01:36.831627 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:36.831570    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr" podStartSLOduration=1.259841419 podStartE2EDuration="1.831555077s" totalImagesPullingTime="571.713658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:36.83037866 +0000 UTC m=+536.398733429" watchObservedRunningTime="2026-11-02 23:01:36.831555077 +0000 UTC m=+536.399909864"
Nov 02 23:01:37.439893 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:37.439852    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:38.966372 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:38.966333    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:38.979352 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:38.979320    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:39.139137 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.139105    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"]
Nov 02 23:01:39.142601 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.142582    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:39.193170 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.193119    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 02 23:01:39.195102 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.195074    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"]
Nov 02 23:01:39.240326 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.240190    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:01:39.249276 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.249239    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20110d77-0721-4910-bcac-42968b4aa72f-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vvvwj\" (UID: \"20110d77-0721-4910-bcac-42968b4aa72f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:39.249436 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.249291    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20110d77-0721-4910-bcac-42968b4aa72f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vvvwj\" (UID: \"20110d77-0721-4910-bcac-42968b4aa72f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:39.249436 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.249373    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhdnf\" (UniqueName: \"kubernetes.io/projected/20110d77-0721-4910-bcac-42968b4aa72f-kube-api-access-bhdnf\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vvvwj\" (UID: \"20110d77-0721-4910-bcac-42968b4aa72f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:39.249850 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.249828    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20110d77-0721-4910-bcac-42968b4aa72f-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vvvwj\" (UID: \"20110d77-0721-4910-bcac-42968b4aa72f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:39.249963 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.249865    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20110d77-0721-4910-bcac-42968b4aa72f-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vvvwj\" (UID: \"20110d77-0721-4910-bcac-42968b4aa72f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:39.249963 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.249892    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20110d77-0721-4910-bcac-42968b4aa72f-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vvvwj\" (UID: \"20110d77-0721-4910-bcac-42968b4aa72f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:39.350683 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.350641    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhdnf\" (UniqueName: \"kubernetes.io/projected/20110d77-0721-4910-bcac-42968b4aa72f-kube-api-access-bhdnf\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vvvwj\" (UID: \"20110d77-0721-4910-bcac-42968b4aa72f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:39.350875 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.350709    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20110d77-0721-4910-bcac-42968b4aa72f-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vvvwj\" (UID: \"20110d77-0721-4910-bcac-42968b4aa72f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:39.350875 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.350731    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20110d77-0721-4910-bcac-42968b4aa72f-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vvvwj\" (UID: \"20110d77-0721-4910-bcac-42968b4aa72f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:39.350875 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.350750    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20110d77-0721-4910-bcac-42968b4aa72f-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vvvwj\" (UID: \"20110d77-0721-4910-bcac-42968b4aa72f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:39.350875 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.350783    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20110d77-0721-4910-bcac-42968b4aa72f-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vvvwj\" (UID: \"20110d77-0721-4910-bcac-42968b4aa72f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:39.350875 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.350828    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20110d77-0721-4910-bcac-42968b4aa72f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vvvwj\" (UID: \"20110d77-0721-4910-bcac-42968b4aa72f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:39.351252 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.351218    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20110d77-0721-4910-bcac-42968b4aa72f-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vvvwj\" (UID: \"20110d77-0721-4910-bcac-42968b4aa72f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:39.351330 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.351264    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20110d77-0721-4910-bcac-42968b4aa72f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vvvwj\" (UID: \"20110d77-0721-4910-bcac-42968b4aa72f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:39.351330 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.351314    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20110d77-0721-4910-bcac-42968b4aa72f-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vvvwj\" (UID: \"20110d77-0721-4910-bcac-42968b4aa72f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:39.353126 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.353106    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20110d77-0721-4910-bcac-42968b4aa72f-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vvvwj\" (UID: \"20110d77-0721-4910-bcac-42968b4aa72f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:39.353389 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.353372    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20110d77-0721-4910-bcac-42968b4aa72f-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vvvwj\" (UID: \"20110d77-0721-4910-bcac-42968b4aa72f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:39.405868 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.405836    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhdnf\" (UniqueName: \"kubernetes.io/projected/20110d77-0721-4910-bcac-42968b4aa72f-kube-api-access-bhdnf\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vvvwj\" (UID: \"20110d77-0721-4910-bcac-42968b4aa72f\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:39.456037 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.456000    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:39.744519 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.744482    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k"
Nov 02 23:01:39.821052 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:39.821027    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"]
Nov 02 23:01:39.822176 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:01:39.822151    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20110d77_0721_4910_bcac_42968b4aa72f.slice/crio-9ee6aee887968608d35f4b36d483b451943b748f30aae0fb3d9ec2ebe015739f WatchSource:0}: Error finding container 9ee6aee887968608d35f4b36d483b451943b748f30aae0fb3d9ec2ebe015739f: Status 404 returned error can't find the container with id 9ee6aee887968608d35f4b36d483b451943b748f30aae0fb3d9ec2ebe015739f
Nov 02 23:01:40.739419 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:40.739320    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj" event={"ID":"20110d77-0721-4910-bcac-42968b4aa72f","Type":"ContainerStarted","Data":"b3ba9ea23f360efcd0e4474f2c812f0b8d92cd0880597651f338bfe6c373b8af"}
Nov 02 23:01:40.739419 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:40.739390    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj" event={"ID":"20110d77-0721-4910-bcac-42968b4aa72f","Type":"ContainerStarted","Data":"9ee6aee887968608d35f4b36d483b451943b748f30aae0fb3d9ec2ebe015739f"}
Nov 02 23:01:40.850446 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:40.850381    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj" podStartSLOduration=1.254659764 podStartE2EDuration="1.850366204s" totalImagesPullingTime="595.70644ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:40.84920668 +0000 UTC m=+540.417561466" watchObservedRunningTime="2026-11-02 23:01:40.850366204 +0000 UTC m=+540.418720990"
Nov 02 23:01:41.456510 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:41.456461    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:41.753861 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:01:41.753829    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:01:42.700018 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.699968    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"]
Nov 02 23:01:42.704698 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.704679    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:42.757411 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.757382    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"]
Nov 02 23:01:42.802598 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.802562    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 02 23:01:42.823756 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.823720    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:01:42.882438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.882397    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/500ba602-25ba-42d0-83ca-a9b8f3bae8fa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kj87x\" (UID: \"500ba602-25ba-42d0-83ca-a9b8f3bae8fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:42.882647 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.882531    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/500ba602-25ba-42d0-83ca-a9b8f3bae8fa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kj87x\" (UID: \"500ba602-25ba-42d0-83ca-a9b8f3bae8fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:42.882647 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.882603    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/500ba602-25ba-42d0-83ca-a9b8f3bae8fa-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kj87x\" (UID: \"500ba602-25ba-42d0-83ca-a9b8f3bae8fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:42.882769 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.882714    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/500ba602-25ba-42d0-83ca-a9b8f3bae8fa-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kj87x\" (UID: \"500ba602-25ba-42d0-83ca-a9b8f3bae8fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:42.882769 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.882748    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sjfc\" (UniqueName: \"kubernetes.io/projected/500ba602-25ba-42d0-83ca-a9b8f3bae8fa-kube-api-access-8sjfc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kj87x\" (UID: \"500ba602-25ba-42d0-83ca-a9b8f3bae8fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:42.882882 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.882854    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/500ba602-25ba-42d0-83ca-a9b8f3bae8fa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kj87x\" (UID: \"500ba602-25ba-42d0-83ca-a9b8f3bae8fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:42.983753 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.983668    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/500ba602-25ba-42d0-83ca-a9b8f3bae8fa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kj87x\" (UID: \"500ba602-25ba-42d0-83ca-a9b8f3bae8fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:42.983753 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.983736    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/500ba602-25ba-42d0-83ca-a9b8f3bae8fa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kj87x\" (UID: \"500ba602-25ba-42d0-83ca-a9b8f3bae8fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:42.984012 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.983758    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/500ba602-25ba-42d0-83ca-a9b8f3bae8fa-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kj87x\" (UID: \"500ba602-25ba-42d0-83ca-a9b8f3bae8fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:42.984012 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.983784    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/500ba602-25ba-42d0-83ca-a9b8f3bae8fa-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kj87x\" (UID: \"500ba602-25ba-42d0-83ca-a9b8f3bae8fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:42.984012 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.983806    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sjfc\" (UniqueName: \"kubernetes.io/projected/500ba602-25ba-42d0-83ca-a9b8f3bae8fa-kube-api-access-8sjfc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kj87x\" (UID: \"500ba602-25ba-42d0-83ca-a9b8f3bae8fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:42.984012 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.983842    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/500ba602-25ba-42d0-83ca-a9b8f3bae8fa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kj87x\" (UID: \"500ba602-25ba-42d0-83ca-a9b8f3bae8fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:42.984229 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.984204    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/500ba602-25ba-42d0-83ca-a9b8f3bae8fa-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kj87x\" (UID: \"500ba602-25ba-42d0-83ca-a9b8f3bae8fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:42.984296 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.984204    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/500ba602-25ba-42d0-83ca-a9b8f3bae8fa-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kj87x\" (UID: \"500ba602-25ba-42d0-83ca-a9b8f3bae8fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:42.984296 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.984264    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/500ba602-25ba-42d0-83ca-a9b8f3bae8fa-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kj87x\" (UID: \"500ba602-25ba-42d0-83ca-a9b8f3bae8fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:42.986211 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.986188    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/500ba602-25ba-42d0-83ca-a9b8f3bae8fa-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kj87x\" (UID: \"500ba602-25ba-42d0-83ca-a9b8f3bae8fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:42.986389 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:42.986369    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/500ba602-25ba-42d0-83ca-a9b8f3bae8fa-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kj87x\" (UID: \"500ba602-25ba-42d0-83ca-a9b8f3bae8fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:43.038456 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:43.038419    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sjfc\" (UniqueName: \"kubernetes.io/projected/500ba602-25ba-42d0-83ca-a9b8f3bae8fa-kube-api-access-8sjfc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-kj87x\" (UID: \"500ba602-25ba-42d0-83ca-a9b8f3bae8fa\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:43.316581 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:43.316539    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:43.685748 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:43.685711    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"]
Nov 02 23:01:43.685880 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:01:43.685801    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500ba602_25ba_42d0_83ca_a9b8f3bae8fa.slice/crio-c1bf40af9059217df8a13682bf0e6a860738226c7d9e4334b0b0164e43f78310 WatchSource:0}: Error finding container c1bf40af9059217df8a13682bf0e6a860738226c7d9e4334b0b0164e43f78310: Status 404 returned error can't find the container with id c1bf40af9059217df8a13682bf0e6a860738226c7d9e4334b0b0164e43f78310
Nov 02 23:01:43.755400 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:43.755364    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x" event={"ID":"500ba602-25ba-42d0-83ca-a9b8f3bae8fa","Type":"ContainerStarted","Data":"c1bf40af9059217df8a13682bf0e6a860738226c7d9e4334b0b0164e43f78310"}
Nov 02 23:01:44.760948 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:44.760908    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x" event={"ID":"500ba602-25ba-42d0-83ca-a9b8f3bae8fa","Type":"ContainerStarted","Data":"b225687e98333bd87d357f31dcadf6d0cca08e387b7ee0c2c3d6b3e4fb0335d3"}
Nov 02 23:01:44.871904 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:44.871844    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x" podStartSLOduration=2.263288829 podStartE2EDuration="2.871828042s" totalImagesPullingTime="608.539213ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:44.870603179 +0000 UTC m=+544.438957965" watchObservedRunningTime="2026-11-02 23:01:44.871828042 +0000 UTC m=+544.440182828"
Nov 02 23:01:45.316966 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:45.316931    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:45.439999 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:45.439956    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:45.452653 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:45.452621    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:45.776595 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:45.776561    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-twdqr"
Nov 02 23:01:47.566695 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:47.566658    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:01:49.456620 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:49.456578    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:49.469505 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:49.469473    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:49.794087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:49.794059    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vvvwj"
Nov 02 23:01:50.668151 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.668115    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-9lmpk"]
Nov 02 23:01:50.671488 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.671469    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:50.723069 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.723036    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 02 23:01:50.725487 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.725459    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-9lmpk"]
Nov 02 23:01:50.754358 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.754318    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b571835c-e34e-4454-8012-31ab1f83b2bc-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-9lmpk\" (UID: \"b571835c-e34e-4454-8012-31ab1f83b2bc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:50.754358 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.754368    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b571835c-e34e-4454-8012-31ab1f83b2bc-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-9lmpk\" (UID: \"b571835c-e34e-4454-8012-31ab1f83b2bc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:50.754571 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.754461    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b571835c-e34e-4454-8012-31ab1f83b2bc-home\") pod \"echo-simulator-kserve-64b6f8f96c-9lmpk\" (UID: \"b571835c-e34e-4454-8012-31ab1f83b2bc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:50.754571 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.754516    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b571835c-e34e-4454-8012-31ab1f83b2bc-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-9lmpk\" (UID: \"b571835c-e34e-4454-8012-31ab1f83b2bc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:50.754571 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.754550    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c88p\" (UniqueName: \"kubernetes.io/projected/b571835c-e34e-4454-8012-31ab1f83b2bc-kube-api-access-9c88p\") pod \"echo-simulator-kserve-64b6f8f96c-9lmpk\" (UID: \"b571835c-e34e-4454-8012-31ab1f83b2bc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:50.754664 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.754602    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b571835c-e34e-4454-8012-31ab1f83b2bc-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-9lmpk\" (UID: \"b571835c-e34e-4454-8012-31ab1f83b2bc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:50.791546 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.791513    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:01:50.855538 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.855500    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b571835c-e34e-4454-8012-31ab1f83b2bc-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-9lmpk\" (UID: \"b571835c-e34e-4454-8012-31ab1f83b2bc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:50.855756 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.855557    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b571835c-e34e-4454-8012-31ab1f83b2bc-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-9lmpk\" (UID: \"b571835c-e34e-4454-8012-31ab1f83b2bc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:50.855756 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.855607    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b571835c-e34e-4454-8012-31ab1f83b2bc-home\") pod \"echo-simulator-kserve-64b6f8f96c-9lmpk\" (UID: \"b571835c-e34e-4454-8012-31ab1f83b2bc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:50.855756 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.855668    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b571835c-e34e-4454-8012-31ab1f83b2bc-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-9lmpk\" (UID: \"b571835c-e34e-4454-8012-31ab1f83b2bc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:50.855756 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.855691    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c88p\" (UniqueName: \"kubernetes.io/projected/b571835c-e34e-4454-8012-31ab1f83b2bc-kube-api-access-9c88p\") pod \"echo-simulator-kserve-64b6f8f96c-9lmpk\" (UID: \"b571835c-e34e-4454-8012-31ab1f83b2bc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:50.856084 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.855866    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b571835c-e34e-4454-8012-31ab1f83b2bc-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-9lmpk\" (UID: \"b571835c-e34e-4454-8012-31ab1f83b2bc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:50.856084 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.856075    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b571835c-e34e-4454-8012-31ab1f83b2bc-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-9lmpk\" (UID: \"b571835c-e34e-4454-8012-31ab1f83b2bc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:50.856420 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.856382    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b571835c-e34e-4454-8012-31ab1f83b2bc-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-9lmpk\" (UID: \"b571835c-e34e-4454-8012-31ab1f83b2bc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:50.856609 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.856587    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b571835c-e34e-4454-8012-31ab1f83b2bc-home\") pod \"echo-simulator-kserve-64b6f8f96c-9lmpk\" (UID: \"b571835c-e34e-4454-8012-31ab1f83b2bc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:50.858017 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.857993    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b571835c-e34e-4454-8012-31ab1f83b2bc-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-9lmpk\" (UID: \"b571835c-e34e-4454-8012-31ab1f83b2bc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:50.858146 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.858130    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b571835c-e34e-4454-8012-31ab1f83b2bc-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-9lmpk\" (UID: \"b571835c-e34e-4454-8012-31ab1f83b2bc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:50.910709 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.910675    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c88p\" (UniqueName: \"kubernetes.io/projected/b571835c-e34e-4454-8012-31ab1f83b2bc-kube-api-access-9c88p\") pod \"echo-simulator-kserve-64b6f8f96c-9lmpk\" (UID: \"b571835c-e34e-4454-8012-31ab1f83b2bc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:50.984300 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:50.984205    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:51.350799 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:51.350767    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-9lmpk"]
Nov 02 23:01:51.351369 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:01:51.351344    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb571835c_e34e_4454_8012_31ab1f83b2bc.slice/crio-b7e4690521b9a8582746caade29d9511b017a1a89a51c01ed336d6ff6448a96b WatchSource:0}: Error finding container b7e4690521b9a8582746caade29d9511b017a1a89a51c01ed336d6ff6448a96b: Status 404 returned error can't find the container with id b7e4690521b9a8582746caade29d9511b017a1a89a51c01ed336d6ff6448a96b
Nov 02 23:01:51.798780 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:51.798733    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk" event={"ID":"b571835c-e34e-4454-8012-31ab1f83b2bc","Type":"ContainerStarted","Data":"b7e4690521b9a8582746caade29d9511b017a1a89a51c01ed336d6ff6448a96b"}
Nov 02 23:01:53.317196 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:53.317147    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:53.330235 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:53.330205    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:53.819942 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:53.819914    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-kj87x"
Nov 02 23:01:54.665439 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:54.665407    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:01:54.826998 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:54.826919    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk" event={"ID":"b571835c-e34e-4454-8012-31ab1f83b2bc","Type":"ContainerStarted","Data":"c5c71918b9aadc2a0cec3f58d8148e0d73aca934dbb52c5213bac17ef478ad7d"}
Nov 02 23:01:54.855957 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:54.855925    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:01:54.938456 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:54.938338    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk" podStartSLOduration=2.502381406 podStartE2EDuration="4.938321769s" totalImagesPullingTime="2.435940363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:01:54.938275883 +0000 UTC m=+554.506630669" watchObservedRunningTime="2026-11-02 23:01:54.938321769 +0000 UTC m=+554.506676558"
Nov 02 23:01:54.984537 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:54.984497    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:01:55.013089 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:55.013055    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:01:58.758332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.758297    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"]
Nov 02 23:01:58.761571 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.761555    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:01:58.815288 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.815256    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"]
Nov 02 23:01:58.828938 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.828904    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f483c007-731c-42e7-a7f2-9b7877fbba41-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j\" (UID: \"f483c007-731c-42e7-a7f2-9b7877fbba41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:01:58.828938 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.828938    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5nlr\" (UniqueName: \"kubernetes.io/projected/f483c007-731c-42e7-a7f2-9b7877fbba41-kube-api-access-l5nlr\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j\" (UID: \"f483c007-731c-42e7-a7f2-9b7877fbba41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:01:58.829157 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.829050    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f483c007-731c-42e7-a7f2-9b7877fbba41-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j\" (UID: \"f483c007-731c-42e7-a7f2-9b7877fbba41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:01:58.829157 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.829100    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f483c007-731c-42e7-a7f2-9b7877fbba41-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j\" (UID: \"f483c007-731c-42e7-a7f2-9b7877fbba41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:01:58.829157 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.829128    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f483c007-731c-42e7-a7f2-9b7877fbba41-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j\" (UID: \"f483c007-731c-42e7-a7f2-9b7877fbba41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:01:58.829277 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.829204    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f483c007-731c-42e7-a7f2-9b7877fbba41-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j\" (UID: \"f483c007-731c-42e7-a7f2-9b7877fbba41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:01:58.859889 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.859844    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 02 23:01:58.877428 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.877396    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:01:58.929941 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.929902    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f483c007-731c-42e7-a7f2-9b7877fbba41-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j\" (UID: \"f483c007-731c-42e7-a7f2-9b7877fbba41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:01:58.930150 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.929960    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f483c007-731c-42e7-a7f2-9b7877fbba41-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j\" (UID: \"f483c007-731c-42e7-a7f2-9b7877fbba41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:01:58.930150 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.930052    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f483c007-731c-42e7-a7f2-9b7877fbba41-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j\" (UID: \"f483c007-731c-42e7-a7f2-9b7877fbba41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:01:58.930150 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.930082    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5nlr\" (UniqueName: \"kubernetes.io/projected/f483c007-731c-42e7-a7f2-9b7877fbba41-kube-api-access-l5nlr\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j\" (UID: \"f483c007-731c-42e7-a7f2-9b7877fbba41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:01:58.930311 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.930159    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f483c007-731c-42e7-a7f2-9b7877fbba41-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j\" (UID: \"f483c007-731c-42e7-a7f2-9b7877fbba41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:01:58.930311 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.930187    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f483c007-731c-42e7-a7f2-9b7877fbba41-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j\" (UID: \"f483c007-731c-42e7-a7f2-9b7877fbba41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:01:58.930430 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.930410    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f483c007-731c-42e7-a7f2-9b7877fbba41-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j\" (UID: \"f483c007-731c-42e7-a7f2-9b7877fbba41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:01:58.930490 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.930466    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f483c007-731c-42e7-a7f2-9b7877fbba41-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j\" (UID: \"f483c007-731c-42e7-a7f2-9b7877fbba41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:01:58.930547 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.930512    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f483c007-731c-42e7-a7f2-9b7877fbba41-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j\" (UID: \"f483c007-731c-42e7-a7f2-9b7877fbba41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:01:58.932421 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.932399    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f483c007-731c-42e7-a7f2-9b7877fbba41-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j\" (UID: \"f483c007-731c-42e7-a7f2-9b7877fbba41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:01:58.932689 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.932669    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f483c007-731c-42e7-a7f2-9b7877fbba41-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j\" (UID: \"f483c007-731c-42e7-a7f2-9b7877fbba41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:01:58.984443 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:58.984411    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5nlr\" (UniqueName: \"kubernetes.io/projected/f483c007-731c-42e7-a7f2-9b7877fbba41-kube-api-access-l5nlr\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j\" (UID: \"f483c007-731c-42e7-a7f2-9b7877fbba41\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:01:59.073500 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:59.073408    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:01:59.434858 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:59.434831    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"]
Nov 02 23:01:59.436504 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:01:59.436479    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf483c007_731c_42e7_a7f2_9b7877fbba41.slice/crio-5135044dde53aa603134bf3c2cbb2b2a5847c19defbc9718bc8be340e5c7ec6b WatchSource:0}: Error finding container 5135044dde53aa603134bf3c2cbb2b2a5847c19defbc9718bc8be340e5c7ec6b: Status 404 returned error can't find the container with id 5135044dde53aa603134bf3c2cbb2b2a5847c19defbc9718bc8be340e5c7ec6b
Nov 02 23:01:59.849134 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:01:59.849099    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j" event={"ID":"f483c007-731c-42e7-a7f2-9b7877fbba41","Type":"ContainerStarted","Data":"5135044dde53aa603134bf3c2cbb2b2a5847c19defbc9718bc8be340e5c7ec6b"}
Nov 02 23:02:00.854570 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:00.854535    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j" event={"ID":"f483c007-731c-42e7-a7f2-9b7877fbba41","Type":"ContainerStarted","Data":"50b569fbc98bfa69ceb05243d4cfb0915fe29e7cd150a1d04c9956c61c85f5e7"}
Nov 02 23:02:00.964504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:00.964453    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j" podStartSLOduration=2.415847866 podStartE2EDuration="2.964437553s" totalImagesPullingTime="548.589687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:00.963258833 +0000 UTC m=+560.531613621" watchObservedRunningTime="2026-11-02 23:02:00.964437553 +0000 UTC m=+560.532792340"
Nov 02 23:02:00.984496 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:00.984459    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:02:00.996973 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:00.996943    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:02:01.074290 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:01.074257    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:02:01.870963 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:01.870933    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-9lmpk"
Nov 02 23:02:09.074275 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:09.074230    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:02:09.086766 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:09.086730    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:02:09.906896 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:09.906866    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:02:11.747635 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:02:11.747596    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:02:12.475839 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.475804    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"]
Nov 02 23:02:12.481430 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.481408    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.531563 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.531531    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tbbgz\""
Nov 02 23:02:12.535601 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.535581    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 23:02:12.562281 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.562242    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.562281 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.562279    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfhgn\" (UniqueName: \"kubernetes.io/projected/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-kube-api-access-vfhgn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.562502 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.562377    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.562502 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.562401    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.562502 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.562421    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.562502 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.562495    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.562679 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.562513    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.562679 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.562533    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.562679 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.562567    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.581729 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.581695    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"]
Nov 02 23:02:12.664170 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.664126    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.664170 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.664174    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfhgn\" (UniqueName: \"kubernetes.io/projected/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-kube-api-access-vfhgn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.664441 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.664265    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.664441 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.664292    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.664441 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.664316    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.664441 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.664394    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.664441 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.664416    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.664775 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.664445    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.664775 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.664489    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.665010 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.664963    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.665140 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.665047    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.665140 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.665126    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.665140 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.665077    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.665414 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.665389    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.667430 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.667412    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.667772 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.667753    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.718632 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.718592    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.718812 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.718765    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfhgn\" (UniqueName: \"kubernetes.io/projected/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-kube-api-access-vfhgn\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sfd24\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:12.794191 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:12.794154    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:13.205144 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:13.205116    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"]
Nov 02 23:02:13.206056 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:02:13.206034    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85cbb9b4_2bbe_41c6_95eb_af57b5ea02b7.slice/crio-c0c1ebec4ea2b9eab4135c6aa17ea384ca74e47ca0a38bc6a149151807e1c178 WatchSource:0}: Error finding container c0c1ebec4ea2b9eab4135c6aa17ea384ca74e47ca0a38bc6a149151807e1c178: Status 404 returned error can't find the container with id c0c1ebec4ea2b9eab4135c6aa17ea384ca74e47ca0a38bc6a149151807e1c178
Nov 02 23:02:13.912505 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:13.912468    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24" event={"ID":"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7","Type":"ContainerStarted","Data":"c0c1ebec4ea2b9eab4135c6aa17ea384ca74e47ca0a38bc6a149151807e1c178"}
Nov 02 23:02:17.420036 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:17.419201    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:02:17.420036 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:17.419384    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:02:17.420036 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:17.419471    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:02:17.933087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:17.933050    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24" event={"ID":"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7","Type":"ContainerStarted","Data":"3047e779ffd360fdf29a143952d9fe45dd30422e7dd7434fe692b6b4776af145"}
Nov 02 23:02:18.094465 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:18.094410    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24" podStartSLOduration=1.884376467 podStartE2EDuration="6.09439758s" totalImagesPullingTime="4.210021113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:18.093019108 +0000 UTC m=+577.661373896" watchObservedRunningTime="2026-11-02 23:02:18.09439758 +0000 UTC m=+577.662752366"
Nov 02 23:02:18.794504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:18.794465    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:18.796114 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:18.796090    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sfd24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 23:02:18.796225 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:18.796138    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24" podUID="85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 23:02:19.795082 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:19.795044    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sfd24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 23:02:19.795515 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:19.795097    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24" podUID="85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 23:02:20.794792 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:20.794759    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sfd24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 23:02:20.795000 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:20.794808    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24" podUID="85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 23:02:21.794467 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:21.794435    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sfd24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 23:02:21.794880 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:21.794494    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24" podUID="85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 23:02:22.794745 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:22.794708    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sfd24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 23:02:22.795194 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:22.794765    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24" podUID="85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 23:02:23.795490 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:23.795399    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sfd24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 23:02:23.795490 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:23.795457    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24" podUID="85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 23:02:24.039787 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:24.039757    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:24.795439 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:24.795400    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sfd24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 02 23:02:24.795622 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:24.795455    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24" podUID="85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 02 23:02:25.387946 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:25.387899    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-575d76f44f-dlvmr"]
Nov 02 23:02:25.395642 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:25.395607    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-575d76f44f-dlvmr"
Nov 02 23:02:25.444257 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:25.444220    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-575d76f44f-dlvmr"]
Nov 02 23:02:25.512153 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:25.512112    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdrb2\" (UniqueName: \"kubernetes.io/projected/0cb7b02e-68fb-4619-a81c-42bcb8113a5c-kube-api-access-zdrb2\") pod \"authorino-575d76f44f-dlvmr\" (UID: \"0cb7b02e-68fb-4619-a81c-42bcb8113a5c\") " pod="kuadrant-system/authorino-575d76f44f-dlvmr"
Nov 02 23:02:25.512364 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:25.512268    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0cb7b02e-68fb-4619-a81c-42bcb8113a5c-tls-cert\") pod \"authorino-575d76f44f-dlvmr\" (UID: \"0cb7b02e-68fb-4619-a81c-42bcb8113a5c\") " pod="kuadrant-system/authorino-575d76f44f-dlvmr"
Nov 02 23:02:25.613680 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:25.613641    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0cb7b02e-68fb-4619-a81c-42bcb8113a5c-tls-cert\") pod \"authorino-575d76f44f-dlvmr\" (UID: \"0cb7b02e-68fb-4619-a81c-42bcb8113a5c\") " pod="kuadrant-system/authorino-575d76f44f-dlvmr"
Nov 02 23:02:25.614011 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:25.613697    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdrb2\" (UniqueName: \"kubernetes.io/projected/0cb7b02e-68fb-4619-a81c-42bcb8113a5c-kube-api-access-zdrb2\") pod \"authorino-575d76f44f-dlvmr\" (UID: \"0cb7b02e-68fb-4619-a81c-42bcb8113a5c\") " pod="kuadrant-system/authorino-575d76f44f-dlvmr"
Nov 02 23:02:25.616846 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:25.616809    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0cb7b02e-68fb-4619-a81c-42bcb8113a5c-tls-cert\") pod \"authorino-575d76f44f-dlvmr\" (UID: \"0cb7b02e-68fb-4619-a81c-42bcb8113a5c\") " pod="kuadrant-system/authorino-575d76f44f-dlvmr"
Nov 02 23:02:25.668352 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:25.668279    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdrb2\" (UniqueName: \"kubernetes.io/projected/0cb7b02e-68fb-4619-a81c-42bcb8113a5c-kube-api-access-zdrb2\") pod \"authorino-575d76f44f-dlvmr\" (UID: \"0cb7b02e-68fb-4619-a81c-42bcb8113a5c\") " pod="kuadrant-system/authorino-575d76f44f-dlvmr"
Nov 02 23:02:25.710672 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:25.710621    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-575d76f44f-dlvmr"
Nov 02 23:02:25.853743 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:25.853709    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:25.854215 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:25.854195    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:25.855193 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:25.855171    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:02:25.944413 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:25.944333    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:26.080337 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:26.080312    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-575d76f44f-dlvmr"]
Nov 02 23:02:26.082605 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:02:26.082573    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cb7b02e_68fb_4619_a81c_42bcb8113a5c.slice/crio-dfef4d353b4dd04debf166050dc83cc9a25557e4cde635440990822e7603b30e WatchSource:0}: Error finding container dfef4d353b4dd04debf166050dc83cc9a25557e4cde635440990822e7603b30e: Status 404 returned error can't find the container with id dfef4d353b4dd04debf166050dc83cc9a25557e4cde635440990822e7603b30e
Nov 02 23:02:26.115483 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:26.115449    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:26.996661 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:26.996622    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-575d76f44f-dlvmr" event={"ID":"0cb7b02e-68fb-4619-a81c-42bcb8113a5c","Type":"ContainerStarted","Data":"dfef4d353b4dd04debf166050dc83cc9a25557e4cde635440990822e7603b30e"}
Nov 02 23:02:27.140919 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:27.140885    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:27.511773 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:27.511737    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:27.697105 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:27.697073    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:28.003294 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:28.003262    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-575d76f44f-dlvmr" event={"ID":"0cb7b02e-68fb-4619-a81c-42bcb8113a5c","Type":"ContainerStarted","Data":"d68b13418c094a725e577b62242576f1f26e64ac4401bca9f445b126d3083249"}
Nov 02 23:02:28.116329 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:28.116276    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-575d76f44f-dlvmr" podStartSLOduration=1.684616205 podStartE2EDuration="3.116261778s" totalImagesPullingTime="1.431645573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:28.115760254 +0000 UTC m=+587.684115042" watchObservedRunningTime="2026-11-02 23:02:28.116261778 +0000 UTC m=+587.684616564"
Nov 02 23:02:28.142123 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:28.142084    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-588dc66bf4-jwdnm"]
Nov 02 23:02:28.142337 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:28.142295    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-588dc66bf4-jwdnm" podUID="44d4e4cc-09eb-4917-9626-a1a7c49935e9" containerName="authorino" containerID="cri-o://a5c3d44f8322d84bdaae05fef221afa55500c80d354b4167e1ea534e208da3d2" gracePeriod=30
Nov 02 23:02:28.390382 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:28.390357    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-588dc66bf4-jwdnm"
Nov 02 23:02:28.541621 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:28.541535    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44d4e4cc-09eb-4917-9626-a1a7c49935e9-tls-cert\" (UniqueName: \"kubernetes.io/secret/44d4e4cc-09eb-4917-9626-a1a7c49935e9-tls-cert\") pod \"44d4e4cc-09eb-4917-9626-a1a7c49935e9\" (UID: \"44d4e4cc-09eb-4917-9626-a1a7c49935e9\") "
Nov 02 23:02:28.541621 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:28.541593    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d4e4cc-09eb-4917-9626-a1a7c49935e9-kube-api-access-pgq5n\" (UniqueName: \"kubernetes.io/projected/44d4e4cc-09eb-4917-9626-a1a7c49935e9-kube-api-access-pgq5n\") pod \"44d4e4cc-09eb-4917-9626-a1a7c49935e9\" (UID: \"44d4e4cc-09eb-4917-9626-a1a7c49935e9\") "
Nov 02 23:02:28.543721 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:28.543681    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d4e4cc-09eb-4917-9626-a1a7c49935e9-kube-api-access-pgq5n" pod "44d4e4cc-09eb-4917-9626-a1a7c49935e9" (UID: "44d4e4cc-09eb-4917-9626-a1a7c49935e9"). InnerVolumeSpecName "kube-api-access-pgq5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:02:28.552638 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:28.552608    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44d4e4cc-09eb-4917-9626-a1a7c49935e9-tls-cert" pod "44d4e4cc-09eb-4917-9626-a1a7c49935e9" (UID: "44d4e4cc-09eb-4917-9626-a1a7c49935e9"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:02:28.651095 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:28.651053    4294 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44d4e4cc-09eb-4917-9626-a1a7c49935e9-tls-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:02:28.651095 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:28.651092    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgq5n\" (UniqueName: \"kubernetes.io/projected/44d4e4cc-09eb-4917-9626-a1a7c49935e9-kube-api-access-pgq5n\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:02:29.009015 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:29.008963    4294 generic.go:356] "Generic (PLEG): container finished" podID="44d4e4cc-09eb-4917-9626-a1a7c49935e9" containerID="a5c3d44f8322d84bdaae05fef221afa55500c80d354b4167e1ea534e208da3d2" exitCode=0
Nov 02 23:02:29.009429 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:29.009034    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-588dc66bf4-jwdnm"
Nov 02 23:02:29.009429 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:29.009054    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-588dc66bf4-jwdnm" event={"ID":"44d4e4cc-09eb-4917-9626-a1a7c49935e9","Type":"ContainerDied","Data":"a5c3d44f8322d84bdaae05fef221afa55500c80d354b4167e1ea534e208da3d2"}
Nov 02 23:02:29.009429 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:29.009110    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-588dc66bf4-jwdnm" event={"ID":"44d4e4cc-09eb-4917-9626-a1a7c49935e9","Type":"ContainerDied","Data":"b9ac1d2f31fcf7aa6be9e07f8978260dfa75c433f067c09a36bdabae50c223c5"}
Nov 02 23:02:29.009429 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:29.009137    4294 scope.go:122] "RemoveContainer" containerID="a5c3d44f8322d84bdaae05fef221afa55500c80d354b4167e1ea534e208da3d2"
Nov 02 23:02:29.018159 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:29.018131    4294 scope.go:122] "RemoveContainer" containerID="a5c3d44f8322d84bdaae05fef221afa55500c80d354b4167e1ea534e208da3d2"
Nov 02 23:02:29.018451 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:02:29.018433    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c3d44f8322d84bdaae05fef221afa55500c80d354b4167e1ea534e208da3d2\": container with ID starting with a5c3d44f8322d84bdaae05fef221afa55500c80d354b4167e1ea534e208da3d2 not found: ID does not exist" containerID="a5c3d44f8322d84bdaae05fef221afa55500c80d354b4167e1ea534e208da3d2"
Nov 02 23:02:29.018520 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:29.018460    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c3d44f8322d84bdaae05fef221afa55500c80d354b4167e1ea534e208da3d2"} err="failed to get container status \"a5c3d44f8322d84bdaae05fef221afa55500c80d354b4167e1ea534e208da3d2\": rpc error: code = NotFound desc = could not find container \"a5c3d44f8322d84bdaae05fef221afa55500c80d354b4167e1ea534e208da3d2\": container with ID starting with a5c3d44f8322d84bdaae05fef221afa55500c80d354b4167e1ea534e208da3d2 not found: ID does not exist"
Nov 02 23:02:29.167915 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:29.167878    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-588dc66bf4-jwdnm"]
Nov 02 23:02:29.218145 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:29.217224    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-588dc66bf4-jwdnm"]
Nov 02 23:02:30.545802 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:30.545770    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d4e4cc-09eb-4917-9626-a1a7c49935e9" path="/var/lib/kubelet/pods/44d4e4cc-09eb-4917-9626-a1a7c49935e9/volumes"
Nov 02 23:02:41.749177 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:02:41.749137    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:02:50.372932 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:50.372900    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:55.557345 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:55.557309    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:55.732786 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:55.732753    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:55.964025 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:55.963920    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:56.120938 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.120906    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:56.192173 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.192141    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"]
Nov 02 23:02:56.192587 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.192574    4294 cpu_manager.go:414] "containerMap: removing container" podUID="44d4e4cc-09eb-4917-9626-a1a7c49935e9" containerName="authorino"
Nov 02 23:02:56.192632 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.192589    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d4e4cc-09eb-4917-9626-a1a7c49935e9" containerName="authorino"
Nov 02 23:02:56.192666 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.192658    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d4e4cc-09eb-4917-9626-a1a7c49935e9" containerName="authorino"
Nov 02 23:02:56.194657 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.194642    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.269514 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.269479    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"]
Nov 02 23:02:56.317826 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.317786    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vph52\" (UniqueName: \"kubernetes.io/projected/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-kube-api-access-vph52\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.318165 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.317830    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-credential-socket\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.318165 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.317906    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-envoy\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.318165 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.317973    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-workload-socket\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.318165 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.318028    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-data\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.318404 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.318178    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-token\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.318404 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.318223    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-podinfo\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.318404 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.318278    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istiod-ca-cert\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.318404 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.318340    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-workload-certs\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.318664 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.318645    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:56.339764 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.339729    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-16aef2f3-openshift-default-dockercfg-jlc75\""
Nov 02 23:02:56.419217 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.419178    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-podinfo\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.419217 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.419218    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istiod-ca-cert\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.419464 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.419265    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-workload-certs\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.419464 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.419313    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vph52\" (UniqueName: \"kubernetes.io/projected/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-kube-api-access-vph52\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.419610 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.419587    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-credential-socket\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.419680 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.419623    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-envoy\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.419680 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.419653    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-workload-socket\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.419790 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.419686    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-data\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.419845 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.419801    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-workload-certs\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.419845 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.419810    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-token\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.420027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.420003    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-credential-socket\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.420200 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.420176    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-data\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.420200 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.420191    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-workload-socket\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.420358 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.420209    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istiod-ca-cert\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.421738 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.421714    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-envoy\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.421822 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.421761    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-podinfo\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.479413 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.479381    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vph52\" (UniqueName: \"kubernetes.io/projected/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-kube-api-access-vph52\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.480850 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.480832    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-token\") pod \"e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") " pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.487846 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.487824    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"]
Nov 02 23:02:56.492358 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.492341    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.499738 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.499713    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:56.506681 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.506663    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:56.552376 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.552345    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-db3e5cc4-openshift-default-dockercfg-vw8jj\""
Nov 02 23:02:56.554537 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.554511    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"]
Nov 02 23:02:56.621721 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.621684    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895718e3-44f1-431d-af81-be3ed2f45b51-istio-podinfo\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.622190 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.621823    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-istio-envoy\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.622190 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.621878    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-workload-certs\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.622190 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.621940    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-istio-data\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.622190 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.622069    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jljkx\" (UniqueName: \"kubernetes.io/projected/895718e3-44f1-431d-af81-be3ed2f45b51-kube-api-access-jljkx\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.622190 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.622167    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-credential-socket\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.622383 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.622242    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895718e3-44f1-431d-af81-be3ed2f45b51-istiod-ca-cert\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.622383 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.622302    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-workload-socket\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.622446 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.622388    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895718e3-44f1-431d-af81-be3ed2f45b51-istio-token\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.666669 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.666635    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:56.723348 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.723296    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-workload-socket\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.723498 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.723362    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895718e3-44f1-431d-af81-be3ed2f45b51-istio-token\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.723498 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.723404    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895718e3-44f1-431d-af81-be3ed2f45b51-istio-podinfo\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.723498 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.723425    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-istio-envoy\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.723498 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.723450    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-workload-certs\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.723498 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.723468    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-istio-data\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.723787 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.723512    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jljkx\" (UniqueName: \"kubernetes.io/projected/895718e3-44f1-431d-af81-be3ed2f45b51-kube-api-access-jljkx\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.723787 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.723573    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-credential-socket\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.723787 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.723604    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895718e3-44f1-431d-af81-be3ed2f45b51-istiod-ca-cert\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.723945 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.723784    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-workload-socket\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.724112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.724089    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-istio-data\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.724253 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.724235    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-workload-certs\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.724353 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.724332    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-credential-socket\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.724722 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.724701    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895718e3-44f1-431d-af81-be3ed2f45b51-istiod-ca-cert\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.725891 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.725865    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-istio-envoy\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.726049 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.726033    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895718e3-44f1-431d-af81-be3ed2f45b51-istio-podinfo\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.796469 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.796403    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895718e3-44f1-431d-af81-be3ed2f45b51-istio-token\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.796601 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.796468    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jljkx\" (UniqueName: \"kubernetes.io/projected/895718e3-44f1-431d-af81-be3ed2f45b51-kube-api-access-jljkx\") pod \"e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") " pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.805512 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.805489    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:56.813325 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.813304    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"]
Nov 02 23:02:56.816888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.816870    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:56.873327 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.873301    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-8a68c823-gw-openshift-default-dockercfg-7nn5j\""
Nov 02 23:02:56.883092 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.883062    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"]
Nov 02 23:02:56.917868 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.917834    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:56.924936 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.924910    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmxdz\" (UniqueName: \"kubernetes.io/projected/ea748a41-7475-4fae-a20e-261cd3e210c7-kube-api-access-lmxdz\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:56.925151 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.924949    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-credential-socket\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:56.925299 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.925221    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-podinfo\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:56.925299 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.925263    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea748a41-7475-4fae-a20e-261cd3e210c7-istiod-ca-cert\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:56.925299 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.925295    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-workload-socket\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:56.925451 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.925317    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-envoy\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:56.925451 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.925362    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-token\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:56.925451 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.925395    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-data\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:56.925451 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.925420    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-workload-certs\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:56.927117 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.927100    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"]
Nov 02 23:02:56.928112 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:02:56.928092    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a60fe3_d55f_4bc2_95bf_64f6fb2b0d00.slice/crio-37909bca7ee622220a5bcbca5faa22f4ea2ac97967ad668dd1cf55872f8708d9 WatchSource:0}: Error finding container 37909bca7ee622220a5bcbca5faa22f4ea2ac97967ad668dd1cf55872f8708d9: Status 404 returned error can't find the container with id 37909bca7ee622220a5bcbca5faa22f4ea2ac97967ad668dd1cf55872f8708d9
Nov 02 23:02:56.930061 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.930034    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:02:56.930122 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.930091    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:02:56.930161 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:56.930119    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:02:57.026654 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.026619    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmxdz\" (UniqueName: \"kubernetes.io/projected/ea748a41-7475-4fae-a20e-261cd3e210c7-kube-api-access-lmxdz\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:57.026807 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.026681    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-credential-socket\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:57.026807 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.026759    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-podinfo\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:57.026807 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.026795    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea748a41-7475-4fae-a20e-261cd3e210c7-istiod-ca-cert\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:57.026949 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.026825    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-workload-socket\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:57.026949 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.026848    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-envoy\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:57.026949 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.026894    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-token\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:57.026949 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.026927    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-data\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:57.027145 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.026951    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-workload-certs\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:57.027520 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.027227    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-credential-socket\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:57.027520 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.027319    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-workload-socket\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:57.027520 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.027363    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-workload-certs\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:57.027758 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.027572    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-data\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:57.027935 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.027917    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea748a41-7475-4fae-a20e-261cd3e210c7-istiod-ca-cert\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:57.030004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.029956    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-podinfo\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:57.030091 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.030010    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-envoy\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:57.086443 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.086365    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-token\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:57.086576 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.086484    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmxdz\" (UniqueName: \"kubernetes.io/projected/ea748a41-7475-4fae-a20e-261cd3e210c7-kube-api-access-lmxdz\") pod \"e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") " pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:57.087723 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.087697    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:57.132148 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.132114    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:57.140400 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.140372    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t" event={"ID":"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00","Type":"ContainerStarted","Data":"1a76b6205325838bec44d3e2c6a2c36e06c8b331e83a224c707a472a649478a3"}
Nov 02 23:02:57.140522 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.140423    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t" event={"ID":"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00","Type":"ContainerStarted","Data":"37909bca7ee622220a5bcbca5faa22f4ea2ac97967ad668dd1cf55872f8708d9"}
Nov 02 23:02:57.193325 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.193296    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"]
Nov 02 23:02:57.195398 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:02:57.195366    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895718e3_44f1_431d_af81_be3ed2f45b51.slice/crio-ef4e8b7229a63017dc9a83e6f667da74a88384bb4fa6c142587ff07e57ba92df WatchSource:0}: Error finding container ef4e8b7229a63017dc9a83e6f667da74a88384bb4fa6c142587ff07e57ba92df: Status 404 returned error can't find the container with id ef4e8b7229a63017dc9a83e6f667da74a88384bb4fa6c142587ff07e57ba92df
Nov 02 23:02:57.198296 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.198266    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:02:57.198395 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.198358    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:02:57.198455 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.198404    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:02:57.256683 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.256628    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t" podStartSLOduration=1.2566076339999999 podStartE2EDuration="1.256607634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:57.254482643 +0000 UTC m=+616.822837456" watchObservedRunningTime="2026-11-02 23:02:57.256607634 +0000 UTC m=+616.824962422"
Nov 02 23:02:57.262510 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.262478    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:57.427371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.427300    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:57.505020 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.504963    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"]
Nov 02 23:02:57.506354 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:02:57.506326    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea748a41_7475_4fae_a20e_261cd3e210c7.slice/crio-b9467c41c049cb9203eede7cb76ac657b98e96a6c2db6850633ff1cee31011dd WatchSource:0}: Error finding container b9467c41c049cb9203eede7cb76ac657b98e96a6c2db6850633ff1cee31011dd: Status 404 returned error can't find the container with id b9467c41c049cb9203eede7cb76ac657b98e96a6c2db6850633ff1cee31011dd
Nov 02 23:02:57.507185 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.507164    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:57.508768 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.508733    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:02:57.508892 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.508799    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:02:57.508892 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.508832    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:02:57.513690 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:57.513659    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:58.147319 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.147281    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6" event={"ID":"ea748a41-7475-4fae-a20e-261cd3e210c7","Type":"ContainerStarted","Data":"1ae33ee115630285528f6d1b9fe4a6fb36b01cac1afa20662508993f7106ff76"}
Nov 02 23:02:58.147775 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.147334    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6" event={"ID":"ea748a41-7475-4fae-a20e-261cd3e210c7","Type":"ContainerStarted","Data":"b9467c41c049cb9203eede7cb76ac657b98e96a6c2db6850633ff1cee31011dd"}
Nov 02 23:02:58.148823 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.148798    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2" event={"ID":"895718e3-44f1-431d-af81-be3ed2f45b51","Type":"ContainerStarted","Data":"319fe6f3a44a883039fd3be1671068f42e560c107beed00fa71acf838a9c5209"}
Nov 02 23:02:58.148913 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.148837    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2" event={"ID":"895718e3-44f1-431d-af81-be3ed2f45b51","Type":"ContainerStarted","Data":"ef4e8b7229a63017dc9a83e6f667da74a88384bb4fa6c142587ff07e57ba92df"}
Nov 02 23:02:58.154760 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.151125    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:58.155916 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.155898    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:02:58.260851 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.260801    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6" podStartSLOduration=2.260788363 podStartE2EDuration="2.260788363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:58.259560644 +0000 UTC m=+617.827915431" watchObservedRunningTime="2026-11-02 23:02:58.260788363 +0000 UTC m=+617.829143150"
Nov 02 23:02:58.372448 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.372398    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2" podStartSLOduration=2.372383979 podStartE2EDuration="2.372383979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:02:58.371972407 +0000 UTC m=+617.940327195" watchObservedRunningTime="2026-11-02 23:02:58.372383979 +0000 UTC m=+617.940738765"
Nov 02 23:02:58.547134 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.547106    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"]
Nov 02 23:02:58.550679 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.550659    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.602937 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.602904    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d0078993-openshift-default-dockercfg-zwqsc\""
Nov 02 23:02:58.606301 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.606273    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"]
Nov 02 23:02:58.644638 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.644601    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-credential-socket\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.644785 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.644672    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3228817-8695-4cea-908c-57cf4bb209bb-istiod-ca-cert\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.644785 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.644750    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-workload-certs\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.644861 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.644827    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3228817-8695-4cea-908c-57cf4bb209bb-istio-token\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.644895 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.644870    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-istio-data\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.644933 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.644912    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-workload-socket\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.645015 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.644998    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t627n\" (UniqueName: \"kubernetes.io/projected/a3228817-8695-4cea-908c-57cf4bb209bb-kube-api-access-t627n\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.645080 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.645064    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3228817-8695-4cea-908c-57cf4bb209bb-istio-podinfo\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.645145 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.645129    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-istio-envoy\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.663648 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.663481    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:58.745679 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.745647    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3228817-8695-4cea-908c-57cf4bb209bb-istio-podinfo\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.745873 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.745693    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-istio-envoy\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.745873 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.745833    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-credential-socket\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.746065 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.745916    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3228817-8695-4cea-908c-57cf4bb209bb-istiod-ca-cert\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.746132 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.746050    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-workload-certs\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.746266 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.746242    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-credential-socket\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.746456 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.746393    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3228817-8695-4cea-908c-57cf4bb209bb-istio-token\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.746456 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.746411    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-workload-certs\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.746456 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.746450    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-istio-data\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.746623 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.746482    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-workload-socket\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.746623 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.746518    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t627n\" (UniqueName: \"kubernetes.io/projected/a3228817-8695-4cea-908c-57cf4bb209bb-kube-api-access-t627n\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.746803 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.746781    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3228817-8695-4cea-908c-57cf4bb209bb-istiod-ca-cert\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.746873 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.746849    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-workload-socket\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.747019 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.746996    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-istio-data\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.748204 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.748184    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-istio-envoy\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.748419 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.748399    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3228817-8695-4cea-908c-57cf4bb209bb-istio-podinfo\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.801337 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.801265    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t627n\" (UniqueName: \"kubernetes.io/projected/a3228817-8695-4cea-908c-57cf4bb209bb-kube-api-access-t627n\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.801920 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.801886    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3228817-8695-4cea-908c-57cf4bb209bb-istio-token\") pod \"e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") " pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:58.807781 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.806539    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:58.813329 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.813303    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:58.855901 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.855871    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:58.862599 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:58.862573    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:02:59.133245 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:59.133177    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:59.138390 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:59.138365    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:59.154314 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:59.154232    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:59.154314 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:59.154291    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:59.155086 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:59.154849    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:02:59.155086 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:59.154894    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:02:59.298819 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:02:59.298787    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3228817_8695_4cea_908c_57cf4bb209bb.slice/crio-fc284eb54de4815dc9bb7ea67a283d1819735039ea02c6da436d36999fd681aa WatchSource:0}: Error finding container fc284eb54de4815dc9bb7ea67a283d1819735039ea02c6da436d36999fd681aa: Status 404 returned error can't find the container with id fc284eb54de4815dc9bb7ea67a283d1819735039ea02c6da436d36999fd681aa
Nov 02 23:02:59.300066 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:59.300042    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"]
Nov 02 23:02:59.301173 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:59.301150    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:02:59.301242 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:59.301203    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:02:59.301242 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:59.301231    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:02:59.495768 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:59.495741    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:59.662088 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:59.662059    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:02:59.836454 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:02:59.836373    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:00.160870 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:00.160779    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd" event={"ID":"a3228817-8695-4cea-908c-57cf4bb209bb","Type":"ContainerStarted","Data":"3a5664377b50ad21bc46e7bbba5468ce85840d3593c20a873746c5015df9c77f"}
Nov 02 23:03:00.160870 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:00.160838    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd" event={"ID":"a3228817-8695-4cea-908c-57cf4bb209bb","Type":"ContainerStarted","Data":"fc284eb54de4815dc9bb7ea67a283d1819735039ea02c6da436d36999fd681aa"}
Nov 02 23:03:00.275229 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:00.275180    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd" podStartSLOduration=2.275166779 podStartE2EDuration="2.275166779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:00.273349551 +0000 UTC m=+619.841704338" watchObservedRunningTime="2026-11-02 23:03:00.275166779 +0000 UTC m=+619.843521640"
Nov 02 23:03:00.709741 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:00.709708    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:00.863589 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:00.863554    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:03:00.868824 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:00.868798    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:03:01.027410 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:01.027371    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:01.165330 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:01.165296    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:03:01.166383 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:01.166364    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:03:01.575775 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:01.575743    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:01.765060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:01.765030    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:02.420020 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:02.419987    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:02.595081 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:02.595042    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:02.774238 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:02.774211    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:03.115949 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.115863    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:03.144739 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.144710    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"]
Nov 02 23:03:03.148513 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.148496    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.198357 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.198324    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-bdaf93e3-openshift-default-dockercfg-bz5st\""
Nov 02 23:03:03.248721 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.248689    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"]
Nov 02 23:03:03.286595 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.286559    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:03.295803 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.295773    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0eade55-f0dd-4879-841b-bb904df6b7d0-istiod-ca-cert\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.295972 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.295818    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-podinfo\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.295972 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.295951    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-envoy\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.296131 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.295975    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-workload-socket\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.296131 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.296068    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pw6f\" (UniqueName: \"kubernetes.io/projected/e0eade55-f0dd-4879-841b-bb904df6b7d0-kube-api-access-2pw6f\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.296234 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.296132    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-credential-socket\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.296234 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.296150    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-workload-certs\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.296348 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.296248    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-data\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.296348 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.296285    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-token\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.396935 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.396836    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-envoy\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.396935 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.396888    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-workload-socket\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.396935 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.396933    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pw6f\" (UniqueName: \"kubernetes.io/projected/e0eade55-f0dd-4879-841b-bb904df6b7d0-kube-api-access-2pw6f\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.397254 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.396975    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-credential-socket\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.397254 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.397026    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-workload-certs\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.397254 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.397076    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-data\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.397254 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.397098    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-token\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.397456 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.397252    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0eade55-f0dd-4879-841b-bb904df6b7d0-istiod-ca-cert\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.397456 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.397327    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-podinfo\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.397456 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.397421    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-credential-socket\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.397605 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.397539    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-data\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.397605 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.397579    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-workload-certs\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.397699 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.397635    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-workload-socket\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.398099 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.398077    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0eade55-f0dd-4879-841b-bb904df6b7d0-istiod-ca-cert\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.399346 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.399326    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-envoy\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.399657 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.399640    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-podinfo\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.460054 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.460022    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-token\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.460392 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.460296    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pw6f\" (UniqueName: \"kubernetes.io/projected/e0eade55-f0dd-4879-841b-bb904df6b7d0-kube-api-access-2pw6f\") pod \"e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") " pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.461098 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.461079    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:03.470637 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.470598    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:03.696316 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.696233    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:03.886048 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.886023    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"]
Nov 02 23:03:03.887157 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:03:03.887131    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0eade55_f0dd_4879_841b_bb904df6b7d0.slice/crio-3a76f1a07b551caad8f77c8a24d5352b627273ee262cdbff3d4b53892bcd74b4 WatchSource:0}: Error finding container 3a76f1a07b551caad8f77c8a24d5352b627273ee262cdbff3d4b53892bcd74b4: Status 404 returned error can't find the container with id 3a76f1a07b551caad8f77c8a24d5352b627273ee262cdbff3d4b53892bcd74b4
Nov 02 23:03:03.889424 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.889392    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:03:03.889513 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.889466    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:03:03.889582 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:03.889510    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:03:04.180437 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:04.180405    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh" event={"ID":"e0eade55-f0dd-4879-841b-bb904df6b7d0","Type":"ContainerStarted","Data":"72bb5df9c45760b9238bedc7712c6e64122c29c6eb3cf37116ba6868aad3d8da"}
Nov 02 23:03:04.180617 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:04.180457    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh" event={"ID":"e0eade55-f0dd-4879-841b-bb904df6b7d0","Type":"ContainerStarted","Data":"3a76f1a07b551caad8f77c8a24d5352b627273ee262cdbff3d4b53892bcd74b4"}
Nov 02 23:03:04.296030 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:04.295950    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh" podStartSLOduration=1.295935551 podStartE2EDuration="1.295935551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:04.295479407 +0000 UTC m=+623.863834193" watchObservedRunningTime="2026-11-02 23:03:04.295935551 +0000 UTC m=+623.864290338"
Nov 02 23:03:04.462116 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:04.462031    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:04.466532 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:04.466507    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:05.185017 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:05.184965    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:05.186091 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:05.186071    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:05.531102 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:05.531071    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:05.703459 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:05.703421    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:05.875260 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:05.875137    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:06.047421 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:06.047355    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:06.205852 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:06.205773    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:07.797261 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:07.797222    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:07.958113 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:07.958080    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:10.118093 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:10.118063    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:10.284872 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:10.284840    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:10.299293 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:10.299262    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"]
Nov 02 23:03:10.303212 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:10.303193    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"
Nov 02 23:03:10.400686 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:10.400615    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d0078993-serving-cert\""
Nov 02 23:03:10.403087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:10.403065    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"]
Nov 02 23:03:10.481918 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:10.481887    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-maas-api-tls\") pod \"maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn\" (UID: \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"
Nov 02 23:03:10.482107 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:10.482055    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqndj\" (UniqueName: \"kubernetes.io/projected/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-kube-api-access-zqndj\") pod \"maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn\" (UID: \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"
Nov 02 23:03:10.482219 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:10.482198    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-metrics-certs\") pod \"maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn\" (UID: \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"
Nov 02 23:03:10.482256 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:10.482233    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-openshift-service-ca\") pod \"maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn\" (UID: \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"
Nov 02 23:03:10.536306 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:10.536271    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:10.584004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:10.583942    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-metrics-certs\") pod \"maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn\" (UID: \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"
Nov 02 23:03:10.584392 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:10.584367    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-openshift-service-ca\") pod \"maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn\" (UID: \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"
Nov 02 23:03:10.585861 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:10.585838    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-maas-api-tls\") pod \"maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn\" (UID: \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"
Nov 02 23:03:10.586767 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:10.586746    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqndj\" (UniqueName: \"kubernetes.io/projected/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-kube-api-access-zqndj\") pod \"maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn\" (UID: \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"
Nov 02 23:03:10.587025 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:03:10.586327    4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-serving-cert: secret "maas-api-e2e-ait-d0078993-serving-cert" not found
Nov 02 23:03:10.587239 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:03:10.587216    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-maas-api-tls podName:7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a nodeName:}" failed. No retries permitted until 2026-11-02 23:03:11.087193432 +0000 UTC m=+630.655548217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-maas-api-tls") pod "maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn" (UID: "7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a") : secret "maas-api-e2e-ait-d0078993-serving-cert" not found
Nov 02 23:03:10.587359 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:10.585458    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-openshift-service-ca\") pod \"maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn\" (UID: \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"
Nov 02 23:03:10.598706 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:10.598683    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-metrics-certs\") pod \"maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn\" (UID: \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"
Nov 02 23:03:10.646939 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:10.646913    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqndj\" (UniqueName: \"kubernetes.io/projected/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-kube-api-access-zqndj\") pod \"maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn\" (UID: \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"
Nov 02 23:03:10.706414 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:10.706344    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:10.880048 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:10.879106    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:11.091248 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:11.091197    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-maas-api-tls\") pod \"maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn\" (UID: \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"
Nov 02 23:03:11.093365 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:11.093341    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-maas-api-tls\") pod \"maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn\" (UID: \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"
Nov 02 23:03:11.216366 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:11.216329    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"
Nov 02 23:03:11.251732 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:11.251705    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:11.428030 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:11.427903    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:11.627796 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:11.627768    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"]
Nov 02 23:03:11.629361 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:03:11.629329    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a5cba77_bc09_4f0e_8aed_d301b1b1dd6a.slice/crio-1b2b40aa426ac6f13ddf2043ce606510dc1dc767dee3b1be0c62a24f785a3373 WatchSource:0}: Error finding container 1b2b40aa426ac6f13ddf2043ce606510dc1dc767dee3b1be0c62a24f785a3373: Status 404 returned error can't find the container with id 1b2b40aa426ac6f13ddf2043ce606510dc1dc767dee3b1be0c62a24f785a3373
Nov 02 23:03:11.693874 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:11.693796    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:11.748651 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:03:11.748617    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:03:12.217780 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:12.217746    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn" event={"ID":"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a","Type":"ContainerStarted","Data":"1b2b40aa426ac6f13ddf2043ce606510dc1dc767dee3b1be0c62a24f785a3373"}
Nov 02 23:03:13.226109 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:13.226068    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn" event={"ID":"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a","Type":"ContainerStarted","Data":"7a20f5c8daacda2fdbc7f96edf79076238c99f9d6e0532381af0538f64689a15"}
Nov 02 23:03:13.226521 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:13.226211    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"
Nov 02 23:03:13.340515 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:13.340453    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn" podStartSLOduration=2.4323319359999998 podStartE2EDuration="3.340440204s" totalImagesPullingTime="908.108268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:13.338822784 +0000 UTC m=+632.907177578" watchObservedRunningTime="2026-11-02 23:03:13.340440204 +0000 UTC m=+632.908794990"
Nov 02 23:03:14.786692 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:14.786642    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:14.988315 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:14.988278    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:15.879518 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:15.879466    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:19.237822 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:19.237788    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"
Nov 02 23:03:20.280299 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.280231    4294 generic.go:356] "Generic (PLEG): container finished" podID="ea748a41-7475-4fae-a20e-261cd3e210c7" containerID="1ae33ee115630285528f6d1b9fe4a6fb36b01cac1afa20662508993f7106ff76" exitCode=0
Nov 02 23:03:20.280778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.280308    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6" event={"ID":"ea748a41-7475-4fae-a20e-261cd3e210c7","Type":"ContainerDied","Data":"1ae33ee115630285528f6d1b9fe4a6fb36b01cac1afa20662508993f7106ff76"}
Nov 02 23:03:20.280845 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.280782    4294 scope.go:122] "RemoveContainer" containerID="1ae33ee115630285528f6d1b9fe4a6fb36b01cac1afa20662508993f7106ff76"
Nov 02 23:03:20.282311 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.282276    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:03:20.282415 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.282367    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:03:20.282415 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.282407    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:03:20.305170 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.305139    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"]
Nov 02 23:03:20.309100 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.309078    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"
Nov 02 23:03:20.387854 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.386274    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-16aef2f3-serving-cert\""
Nov 02 23:03:20.388882 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.388858    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"]
Nov 02 23:03:20.394538 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.393654    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9cef540-e743-42c9-8904-181c1d1c0695-maas-api-tls\") pod \"maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q\" (UID: \"d9cef540-e743-42c9-8904-181c1d1c0695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"
Nov 02 23:03:20.394538 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.393719    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmkjn\" (UniqueName: \"kubernetes.io/projected/d9cef540-e743-42c9-8904-181c1d1c0695-kube-api-access-fmkjn\") pod \"maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q\" (UID: \"d9cef540-e743-42c9-8904-181c1d1c0695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"
Nov 02 23:03:20.394538 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.393816    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9cef540-e743-42c9-8904-181c1d1c0695-metrics-certs\") pod \"maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q\" (UID: \"d9cef540-e743-42c9-8904-181c1d1c0695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"
Nov 02 23:03:20.394538 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.393850    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9cef540-e743-42c9-8904-181c1d1c0695-openshift-service-ca\") pod \"maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q\" (UID: \"d9cef540-e743-42c9-8904-181c1d1c0695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"
Nov 02 23:03:20.495702 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.495371    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9cef540-e743-42c9-8904-181c1d1c0695-metrics-certs\") pod \"maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q\" (UID: \"d9cef540-e743-42c9-8904-181c1d1c0695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"
Nov 02 23:03:20.495702 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.495442    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9cef540-e743-42c9-8904-181c1d1c0695-openshift-service-ca\") pod \"maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q\" (UID: \"d9cef540-e743-42c9-8904-181c1d1c0695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"
Nov 02 23:03:20.495702 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.495494    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9cef540-e743-42c9-8904-181c1d1c0695-maas-api-tls\") pod \"maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q\" (UID: \"d9cef540-e743-42c9-8904-181c1d1c0695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"
Nov 02 23:03:20.495702 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.495563    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmkjn\" (UniqueName: \"kubernetes.io/projected/d9cef540-e743-42c9-8904-181c1d1c0695-kube-api-access-fmkjn\") pod \"maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q\" (UID: \"d9cef540-e743-42c9-8904-181c1d1c0695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"
Nov 02 23:03:20.496073 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:03:20.495995    4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-serving-cert: secret "maas-api-e2e-ait-16aef2f3-serving-cert" not found
Nov 02 23:03:20.496132 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:03:20.496088    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9cef540-e743-42c9-8904-181c1d1c0695-maas-api-tls podName:d9cef540-e743-42c9-8904-181c1d1c0695 nodeName:}" failed. No retries permitted until 2026-11-02 23:03:20.996065568 +0000 UTC m=+640.564420347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d9cef540-e743-42c9-8904-181c1d1c0695-maas-api-tls") pod "maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q" (UID: "d9cef540-e743-42c9-8904-181c1d1c0695") : secret "maas-api-e2e-ait-16aef2f3-serving-cert" not found
Nov 02 23:03:20.496920 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.496895    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9cef540-e743-42c9-8904-181c1d1c0695-openshift-service-ca\") pod \"maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q\" (UID: \"d9cef540-e743-42c9-8904-181c1d1c0695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"
Nov 02 23:03:20.498290 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.498226    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9cef540-e743-42c9-8904-181c1d1c0695-metrics-certs\") pod \"maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q\" (UID: \"d9cef540-e743-42c9-8904-181c1d1c0695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"
Nov 02 23:03:20.521576 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.521518    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:20.550678 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.550598    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmkjn\" (UniqueName: \"kubernetes.io/projected/d9cef540-e743-42c9-8904-181c1d1c0695-kube-api-access-fmkjn\") pod \"maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q\" (UID: \"d9cef540-e743-42c9-8904-181c1d1c0695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"
Nov 02 23:03:20.680524 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:20.680468    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:21.004320 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:21.004272    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9cef540-e743-42c9-8904-181c1d1c0695-maas-api-tls\") pod \"maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q\" (UID: \"d9cef540-e743-42c9-8904-181c1d1c0695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"
Nov 02 23:03:21.014630 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:21.013603    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9cef540-e743-42c9-8904-181c1d1c0695-maas-api-tls\") pod \"maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q\" (UID: \"d9cef540-e743-42c9-8904-181c1d1c0695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"
Nov 02 23:03:21.237548 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:21.237511    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"
Nov 02 23:03:21.309367 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:21.309258    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6" event={"ID":"ea748a41-7475-4fae-a20e-261cd3e210c7","Type":"ContainerStarted","Data":"1ffbbf6dbf10cd518e0684858c1797ca8684e7da773d2296fa31ff4055909e36"}
Nov 02 23:03:21.697181 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:21.697071    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"]
Nov 02 23:03:21.705638 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:21.704876    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:21.705638 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:03:21.705502    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9cef540_e743_42c9_8904_181c1d1c0695.slice/crio-d0d87e21e89d28ee0f369f639481a49a0571506cc1db5a4bf54dc7719e104210 WatchSource:0}: Error finding container d0d87e21e89d28ee0f369f639481a49a0571506cc1db5a4bf54dc7719e104210: Status 404 returned error can't find the container with id d0d87e21e89d28ee0f369f639481a49a0571506cc1db5a4bf54dc7719e104210
Nov 02 23:03:21.876010 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:21.873093    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:22.133366 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:22.133331    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:03:22.133567 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:22.133462    4294 patch_prober.go:29] interesting pod/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 02 23:03:22.133567 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:22.133506    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6" podUID="ea748a41-7475-4fae-a20e-261cd3e210c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 02 23:03:22.315391 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:22.315351    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q" event={"ID":"d9cef540-e743-42c9-8904-181c1d1c0695","Type":"ContainerStarted","Data":"d0d87e21e89d28ee0f369f639481a49a0571506cc1db5a4bf54dc7719e104210"}
Nov 02 23:03:23.133370 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:23.133335    4294 patch_prober.go:29] interesting pod/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 02 23:03:23.133558 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:23.133392    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6" podUID="ea748a41-7475-4fae-a20e-261cd3e210c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 02 23:03:23.322884 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:23.322845    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q" event={"ID":"d9cef540-e743-42c9-8904-181c1d1c0695","Type":"ContainerStarted","Data":"0e9260370b1068cc3de7483e812257d5e98ff2b1aaee514f36e0f463034b644a"}
Nov 02 23:03:23.323718 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:23.323696    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"
Nov 02 23:03:23.442157 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:23.442057    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q" podStartSLOduration=2.517170626 podStartE2EDuration="3.442041867s" totalImagesPullingTime="924.871241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:23.441190368 +0000 UTC m=+643.009545156" watchObservedRunningTime="2026-11-02 23:03:23.442041867 +0000 UTC m=+643.010396686"
Nov 02 23:03:24.133172 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:24.133138    4294 patch_prober.go:29] interesting pod/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 02 23:03:24.133354 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:24.133189    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6" podUID="ea748a41-7475-4fae-a20e-261cd3e210c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 02 23:03:24.411441 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:24.411358    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:24.997379 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:24.997336    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"]
Nov 02 23:03:25.001915 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.001879    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"
Nov 02 23:03:25.031567 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.031538    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"]
Nov 02 23:03:25.051042 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.051013    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lkk2\" (UniqueName: \"kubernetes.io/projected/6901cec5-f339-4eca-8a07-70f3bdadd6cf-kube-api-access-5lkk2\") pod \"maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6\" (UID: \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"
Nov 02 23:03:25.051195 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.051057    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6901cec5-f339-4eca-8a07-70f3bdadd6cf-metrics-certs\") pod \"maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6\" (UID: \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"
Nov 02 23:03:25.051195 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.051151    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6901cec5-f339-4eca-8a07-70f3bdadd6cf-maas-api-tls\") pod \"maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6\" (UID: \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"
Nov 02 23:03:25.051293 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.051276    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6901cec5-f339-4eca-8a07-70f3bdadd6cf-openshift-service-ca\") pod \"maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6\" (UID: \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"
Nov 02 23:03:25.078272 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.078224    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-bdaf93e3-serving-cert\""
Nov 02 23:03:25.133571 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.133545    4294 patch_prober.go:29] interesting pod/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 02 23:03:25.133681 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.133598    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6" podUID="ea748a41-7475-4fae-a20e-261cd3e210c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 02 23:03:25.152431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.152397    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6901cec5-f339-4eca-8a07-70f3bdadd6cf-openshift-service-ca\") pod \"maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6\" (UID: \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"
Nov 02 23:03:25.152550 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.152465    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lkk2\" (UniqueName: \"kubernetes.io/projected/6901cec5-f339-4eca-8a07-70f3bdadd6cf-kube-api-access-5lkk2\") pod \"maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6\" (UID: \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"
Nov 02 23:03:25.152550 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.152512    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6901cec5-f339-4eca-8a07-70f3bdadd6cf-metrics-certs\") pod \"maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6\" (UID: \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"
Nov 02 23:03:25.152625 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.152584    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6901cec5-f339-4eca-8a07-70f3bdadd6cf-maas-api-tls\") pod \"maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6\" (UID: \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"
Nov 02 23:03:25.152762 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:03:25.152747    4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-serving-cert: secret "maas-api-e2e-derive-bdaf93e3-serving-cert" not found
Nov 02 23:03:25.152830 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:03:25.152819    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6901cec5-f339-4eca-8a07-70f3bdadd6cf-maas-api-tls podName:6901cec5-f339-4eca-8a07-70f3bdadd6cf nodeName:}" failed. No retries permitted until 2026-11-02 23:03:25.652798026 +0000 UTC m=+645.221152805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6901cec5-f339-4eca-8a07-70f3bdadd6cf-maas-api-tls") pod "maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6" (UID: "6901cec5-f339-4eca-8a07-70f3bdadd6cf") : secret "maas-api-e2e-derive-bdaf93e3-serving-cert" not found
Nov 02 23:03:25.176003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.153181    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6901cec5-f339-4eca-8a07-70f3bdadd6cf-openshift-service-ca\") pod \"maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6\" (UID: \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"
Nov 02 23:03:25.176003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.175750    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6901cec5-f339-4eca-8a07-70f3bdadd6cf-metrics-certs\") pod \"maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6\" (UID: \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"
Nov 02 23:03:25.195763 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.195732    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:25.207972 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.207945    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lkk2\" (UniqueName: \"kubernetes.io/projected/6901cec5-f339-4eca-8a07-70f3bdadd6cf-kube-api-access-5lkk2\") pod \"maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6\" (UID: \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"
Nov 02 23:03:25.351822 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.351734    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:25.689714 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.689622    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6901cec5-f339-4eca-8a07-70f3bdadd6cf-maas-api-tls\") pod \"maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6\" (UID: \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"
Nov 02 23:03:25.692226 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.692201    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6901cec5-f339-4eca-8a07-70f3bdadd6cf-maas-api-tls\") pod \"maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6\" (UID: \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"
Nov 02 23:03:25.914171 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:25.914137    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"
Nov 02 23:03:26.133135 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:26.133096    4294 patch_prober.go:29] interesting pod/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Nov 02 23:03:26.133306 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:26.133173    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6" podUID="ea748a41-7475-4fae-a20e-261cd3e210c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Nov 02 23:03:26.307094 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:26.307070    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"]
Nov 02 23:03:26.308697 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:03:26.308668    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6901cec5_f339_4eca_8a07_70f3bdadd6cf.slice/crio-934f2c2fe0e2a9036ad80b31c3b066cde3041fa0502fb0b652df117272757f26 WatchSource:0}: Error finding container 934f2c2fe0e2a9036ad80b31c3b066cde3041fa0502fb0b652df117272757f26: Status 404 returned error can't find the container with id 934f2c2fe0e2a9036ad80b31c3b066cde3041fa0502fb0b652df117272757f26
Nov 02 23:03:26.338448 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:26.338416    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6" event={"ID":"6901cec5-f339-4eca-8a07-70f3bdadd6cf","Type":"ContainerStarted","Data":"934f2c2fe0e2a9036ad80b31c3b066cde3041fa0502fb0b652df117272757f26"}
Nov 02 23:03:27.136861 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:27.136824    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:03:27.137505 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:27.137281    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:03:27.138017 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:27.137997    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:03:27.345613 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:27.345576    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6" event={"ID":"6901cec5-f339-4eca-8a07-70f3bdadd6cf","Type":"ContainerStarted","Data":"cca1b030578bb0f7dd049e113ff13b97026a60f16dc006ab5c0b26cdcfeaf790"}
Nov 02 23:03:27.560749 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:27.560692    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6" podStartSLOduration=2.639917665 podStartE2EDuration="3.560677025s" totalImagesPullingTime="920.75936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:27.559108078 +0000 UTC m=+647.127462868" watchObservedRunningTime="2026-11-02 23:03:27.560677025 +0000 UTC m=+647.129031811"
Nov 02 23:03:28.350834 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:28.350800    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"
Nov 02 23:03:29.357223 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:29.357185    4294 generic.go:356] "Generic (PLEG): container finished" podID="a3228817-8695-4cea-908c-57cf4bb209bb" containerID="3a5664377b50ad21bc46e7bbba5468ce85840d3593c20a873746c5015df9c77f" exitCode=0
Nov 02 23:03:29.357685 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:29.357261    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd" event={"ID":"a3228817-8695-4cea-908c-57cf4bb209bb","Type":"ContainerDied","Data":"3a5664377b50ad21bc46e7bbba5468ce85840d3593c20a873746c5015df9c77f"}
Nov 02 23:03:29.357800 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:29.357779    4294 scope.go:122] "RemoveContainer" containerID="3a5664377b50ad21bc46e7bbba5468ce85840d3593c20a873746c5015df9c77f"
Nov 02 23:03:29.359113 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:29.359085    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:03:29.359209 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:29.359163    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:03:29.359281 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:29.359212    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:03:29.974184 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:29.974147    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:30.338632 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:30.338603    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"
Nov 02 23:03:30.383898 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:30.383863    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd" event={"ID":"a3228817-8695-4cea-908c-57cf4bb209bb","Type":"ContainerStarted","Data":"088ddad49b172503a75cf5ea738a3d5fcff0c07857cc7ad4c2cc207a46057612"}
Nov 02 23:03:30.876003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:30.875478    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:03:30.878106 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:30.878072    4294 patch_prober.go:29] interesting pod/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 02 23:03:30.878205 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:30.878128    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd" podUID="a3228817-8695-4cea-908c-57cf4bb209bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 02 23:03:31.171060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:31.170086    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:03:31.872006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:31.871131    4294 patch_prober.go:29] interesting pod/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 02 23:03:31.872006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:31.871193    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd" podUID="a3228817-8695-4cea-908c-57cf4bb209bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 02 23:03:32.877002 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:32.875342    4294 patch_prober.go:29] interesting pod/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 02 23:03:32.877002 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:32.875401    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd" podUID="a3228817-8695-4cea-908c-57cf4bb209bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 02 23:03:33.870004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:33.865351    4294 patch_prober.go:29] interesting pod/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 02 23:03:33.870004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:33.865412    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd" podUID="a3228817-8695-4cea-908c-57cf4bb209bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 02 23:03:33.882048 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:33.882005    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"]
Nov 02 23:03:33.885514 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:33.885488    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"
Nov 02 23:03:33.935879 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:33.935850    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"]
Nov 02 23:03:33.978362 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:33.978273    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-db3e5cc4-serving-cert\""
Nov 02 23:03:34.007382 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:34.007347    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf27287-71e0-48f6-92d0-3cc184d00e2f-metrics-certs\") pod \"maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b\" (UID: \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"
Nov 02 23:03:34.007635 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:34.007604    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv8kt\" (UniqueName: \"kubernetes.io/projected/ddf27287-71e0-48f6-92d0-3cc184d00e2f-kube-api-access-nv8kt\") pod \"maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b\" (UID: \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"
Nov 02 23:03:34.007763 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:34.007725    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf27287-71e0-48f6-92d0-3cc184d00e2f-openshift-service-ca\") pod \"maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b\" (UID: \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"
Nov 02 23:03:34.007828 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:34.007785    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf27287-71e0-48f6-92d0-3cc184d00e2f-maas-api-tls\") pod \"maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b\" (UID: \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"
Nov 02 23:03:34.108518 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:34.108480    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf27287-71e0-48f6-92d0-3cc184d00e2f-metrics-certs\") pod \"maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b\" (UID: \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"
Nov 02 23:03:34.108518 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:34.108518    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv8kt\" (UniqueName: \"kubernetes.io/projected/ddf27287-71e0-48f6-92d0-3cc184d00e2f-kube-api-access-nv8kt\") pod \"maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b\" (UID: \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"
Nov 02 23:03:34.108721 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:34.108572    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf27287-71e0-48f6-92d0-3cc184d00e2f-openshift-service-ca\") pod \"maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b\" (UID: \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"
Nov 02 23:03:34.108721 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:34.108612    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf27287-71e0-48f6-92d0-3cc184d00e2f-maas-api-tls\") pod \"maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b\" (UID: \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"
Nov 02 23:03:34.108793 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:03:34.108757    4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-serving-cert: secret "maas-api-e2e-ait-db3e5cc4-serving-cert" not found
Nov 02 23:03:34.108827 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:03:34.108806    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddf27287-71e0-48f6-92d0-3cc184d00e2f-maas-api-tls podName:ddf27287-71e0-48f6-92d0-3cc184d00e2f nodeName:}" failed. No retries permitted until 2026-11-02 23:03:34.608791519 +0000 UTC m=+654.177146297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ddf27287-71e0-48f6-92d0-3cc184d00e2f-maas-api-tls") pod "maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b" (UID: "ddf27287-71e0-48f6-92d0-3cc184d00e2f") : secret "maas-api-e2e-ait-db3e5cc4-serving-cert" not found
Nov 02 23:03:34.109446 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:34.109424    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf27287-71e0-48f6-92d0-3cc184d00e2f-openshift-service-ca\") pod \"maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b\" (UID: \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"
Nov 02 23:03:34.110817 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:34.110801    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf27287-71e0-48f6-92d0-3cc184d00e2f-metrics-certs\") pod \"maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b\" (UID: \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"
Nov 02 23:03:34.166601 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:34.166516    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:34.178710 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:34.178659    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv8kt\" (UniqueName: \"kubernetes.io/projected/ddf27287-71e0-48f6-92d0-3cc184d00e2f-kube-api-access-nv8kt\") pod \"maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b\" (UID: \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"
Nov 02 23:03:34.336943 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:34.336911    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:34.381548 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:34.381518    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"
Nov 02 23:03:34.508315 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:34.508284    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:34.614773 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:34.614729    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf27287-71e0-48f6-92d0-3cc184d00e2f-maas-api-tls\") pod \"maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b\" (UID: \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"
Nov 02 23:03:34.616971 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:34.616952    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf27287-71e0-48f6-92d0-3cc184d00e2f-maas-api-tls\") pod \"maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b\" (UID: \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"
Nov 02 23:03:34.797511 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:34.797432    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"
Nov 02 23:03:34.867000 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:34.863369    4294 patch_prober.go:29] interesting pod/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 02 23:03:34.867000 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:34.863428    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd" podUID="a3228817-8695-4cea-908c-57cf4bb209bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 02 23:03:35.179932 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:35.179823    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"]
Nov 02 23:03:35.184038 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:03:35.184003    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf27287_71e0_48f6_92d0_3cc184d00e2f.slice/crio-61d3a0bae6d9fc0f1e98e46923071e6e3c57021bb02c5ff1ce0fa1d9ca32fb12 WatchSource:0}: Error finding container 61d3a0bae6d9fc0f1e98e46923071e6e3c57021bb02c5ff1ce0fa1d9ca32fb12: Status 404 returned error can't find the container with id 61d3a0bae6d9fc0f1e98e46923071e6e3c57021bb02c5ff1ce0fa1d9ca32fb12
Nov 02 23:03:35.409756 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:35.409677    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b" event={"ID":"ddf27287-71e0-48f6-92d0-3cc184d00e2f","Type":"ContainerStarted","Data":"61d3a0bae6d9fc0f1e98e46923071e6e3c57021bb02c5ff1ce0fa1d9ca32fb12"}
Nov 02 23:03:35.867314 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:35.867284    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:03:35.868137 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:35.868115    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:03:36.416750 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:36.416660    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b" event={"ID":"ddf27287-71e0-48f6-92d0-3cc184d00e2f","Type":"ContainerStarted","Data":"21a0932ee8acd93adea8db8a7b575c9aa1b212f436fc1f8911357f3a035a3175"}
Nov 02 23:03:36.417234 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:36.416924    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"
Nov 02 23:03:36.529244 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:36.529178    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b" podStartSLOduration=2.557612039 podStartE2EDuration="3.52915461s" totalImagesPullingTime="971.542571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:03:36.525867197 +0000 UTC m=+656.094221986" watchObservedRunningTime="2026-11-02 23:03:36.52915461 +0000 UTC m=+656.097509398"
Nov 02 23:03:39.137355 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:39.137322    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:41.748489 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:03:41.748454    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:03:42.426688 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:42.426662    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"
Nov 02 23:03:43.663906 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:43.663863    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:43.832608 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:43.832568    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:46.490612 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:46.490585    4294 generic.go:356] "Generic (PLEG): container finished" podID="e0eade55-f0dd-4879-841b-bb904df6b7d0" containerID="72bb5df9c45760b9238bedc7712c6e64122c29c6eb3cf37116ba6868aad3d8da" exitCode=0
Nov 02 23:03:46.491053 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:46.490661    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh" event={"ID":"e0eade55-f0dd-4879-841b-bb904df6b7d0","Type":"ContainerDied","Data":"72bb5df9c45760b9238bedc7712c6e64122c29c6eb3cf37116ba6868aad3d8da"}
Nov 02 23:03:46.491053 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:46.491033    4294 scope.go:122] "RemoveContainer" containerID="72bb5df9c45760b9238bedc7712c6e64122c29c6eb3cf37116ba6868aad3d8da"
Nov 02 23:03:46.492250 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:46.492220    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:03:46.492365 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:46.492303    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:03:46.492365 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:46.492351    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:03:47.505744 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:47.505707    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh" event={"ID":"e0eade55-f0dd-4879-841b-bb904df6b7d0","Type":"ContainerStarted","Data":"7608858135a9b5265a8aa0733366fcbeaa5f23607d921d0bcb22b177dfd37206"}
Nov 02 23:03:47.947201 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:47.947118    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:48.122384 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:48.122349    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:48.466004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:48.462121    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:48.466004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:48.463724    4294 patch_prober.go:29] interesting pod/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 02 23:03:48.466004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:48.463802    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh" podUID="e0eade55-f0dd-4879-841b-bb904df6b7d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 02 23:03:48.687752 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:48.687720    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:49.473399 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:49.473360    4294 patch_prober.go:29] interesting pod/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 02 23:03:49.474408 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:49.474375    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh" podUID="e0eade55-f0dd-4879-841b-bb904df6b7d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 02 23:03:50.186002 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:50.185086    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:50.467070 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:50.466386    4294 patch_prober.go:29] interesting pod/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 02 23:03:50.467070 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:50.466464    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh" podUID="e0eade55-f0dd-4879-841b-bb904df6b7d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 02 23:03:51.477003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:51.470333    4294 patch_prober.go:29] interesting pod/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 02 23:03:51.477003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:51.470395    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh" podUID="e0eade55-f0dd-4879-841b-bb904df6b7d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 02 23:03:52.464451 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:52.464405    4294 patch_prober.go:29] interesting pod/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Nov 02 23:03:52.464758 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:52.464730    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh" podUID="e0eade55-f0dd-4879-841b-bb904df6b7d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Nov 02 23:03:53.474004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:53.473388    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:53.479253 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:53.479196    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:03:53.687814 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:53.687479    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:55.551600 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:55.551565    4294 generic.go:356] "Generic (PLEG): container finished" podID="895718e3-44f1-431d-af81-be3ed2f45b51" containerID="319fe6f3a44a883039fd3be1671068f42e560c107beed00fa71acf838a9c5209" exitCode=0
Nov 02 23:03:55.552068 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:55.551639    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2" event={"ID":"895718e3-44f1-431d-af81-be3ed2f45b51","Type":"ContainerDied","Data":"319fe6f3a44a883039fd3be1671068f42e560c107beed00fa71acf838a9c5209"}
Nov 02 23:03:55.552068 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:55.552058    4294 scope.go:122] "RemoveContainer" containerID="319fe6f3a44a883039fd3be1671068f42e560c107beed00fa71acf838a9c5209"
Nov 02 23:03:55.553514 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:55.553479    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:03:55.553759 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:55.553735    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:03:55.553959 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:55.553936    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:03:56.606007 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:56.605954    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2" event={"ID":"895718e3-44f1-431d-af81-be3ed2f45b51","Type":"ContainerStarted","Data":"29eee990be6bb8d3d59343c0bec4f7db74f68b62119a4706e66a24c94289abbc"}
Nov 02 23:03:56.807153 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:56.807106    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:03:56.808523 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:56.808488    4294 patch_prober.go:29] interesting pod/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 02 23:03:56.808706 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:56.808548    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2" podUID="895718e3-44f1-431d-af81-be3ed2f45b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 02 23:03:57.812496 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:57.812458    4294 patch_prober.go:29] interesting pod/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 02 23:03:57.813241 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:57.813207    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2" podUID="895718e3-44f1-431d-af81-be3ed2f45b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 02 23:03:58.010936 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:58.010896    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:58.805878 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:58.805840    4294 patch_prober.go:29] interesting pod/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 02 23:03:58.806076 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:58.805905    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2" podUID="895718e3-44f1-431d-af81-be3ed2f45b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 02 23:03:59.174006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:59.173088    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:03:59.587899 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:59.587856    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:59.769577 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:59.769502    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:03:59.806749 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:59.806710    4294 patch_prober.go:29] interesting pod/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 02 23:03:59.806916 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:59.806792    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2" podUID="895718e3-44f1-431d-af81-be3ed2f45b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 02 23:03:59.935303 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:03:59.935217    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:00.181369 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:00.181329    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"]
Nov 02 23:04:00.181893 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:00.181715    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6" podUID="ea748a41-7475-4fae-a20e-261cd3e210c7" containerName="istio-proxy" containerID="cri-o://1ffbbf6dbf10cd518e0684858c1797ca8684e7da773d2296fa31ff4055909e36" gracePeriod=30
Nov 02 23:04:00.218545 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:00.218450    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:00.579753 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:00.579717    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:00.768817 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:00.768766    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:00.806622 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:00.806585    4294 patch_prober.go:29] interesting pod/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 02 23:04:00.806794 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:00.806654    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2" podUID="895718e3-44f1-431d-af81-be3ed2f45b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 02 23:04:00.979681 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:00.979560    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:01.540729 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:01.540679    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:01.928584 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:01.928496    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:01.931753 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:01.931725    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:04:01.933140 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:01.933117    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:04:03.644412 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:03.644374    4294 generic.go:356] "Generic (PLEG): container finished" podID="57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" containerID="1a76b6205325838bec44d3e2c6a2c36e06c8b331e83a224c707a472a649478a3" exitCode=0
Nov 02 23:04:03.644864 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:03.644445    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t" event={"ID":"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00","Type":"ContainerDied","Data":"1a76b6205325838bec44d3e2c6a2c36e06c8b331e83a224c707a472a649478a3"}
Nov 02 23:04:03.644864 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:03.644838    4294 scope.go:122] "RemoveContainer" containerID="1a76b6205325838bec44d3e2c6a2c36e06c8b331e83a224c707a472a649478a3"
Nov 02 23:04:03.646112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:03.646086    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:04:03.646174 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:03.646153    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:04:03.646209 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:03.646183    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:04:03.852090 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:03.852051    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"]
Nov 02 23:04:03.852327 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:03.852300    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn" podUID="7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a" containerName="maas-api" containerID="cri-o://7a20f5c8daacda2fdbc7f96edf79076238c99f9d6e0532381af0538f64689a15" gracePeriod=30
Nov 02 23:04:03.983599 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:03.983569    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:04.162232 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.162194    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"
Nov 02 23:04:04.217149 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.217116    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-kube-api-access-zqndj\" (UniqueName: \"kubernetes.io/projected/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-kube-api-access-zqndj\") pod \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\" (UID: \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\") "
Nov 02 23:04:04.217329 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.217269    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-maas-api-tls\") pod \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\" (UID: \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\") "
Nov 02 23:04:04.217329 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.217310    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-metrics-certs\") pod \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\" (UID: \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\") "
Nov 02 23:04:04.217429 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.217347    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-openshift-service-ca\") pod \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\" (UID: \"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a\") "
Nov 02 23:04:04.217775 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.217745    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-openshift-service-ca" pod "7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a" (UID: "7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:04:04.219428 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.219367    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-maas-api-tls" pod "7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a" (UID: "7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:04:04.219526 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.219450    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-kube-api-access-zqndj" pod "7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a" (UID: "7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a"). InnerVolumeSpecName "kube-api-access-zqndj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:04:04.219526 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.219487    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-metrics-certs" pod "7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a" (UID: "7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:04:04.318737 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.318694    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:04.318737 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.318730    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:04.318737 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.318740    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:04.318737 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.318749    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqndj\" (UniqueName: \"kubernetes.io/projected/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a-kube-api-access-zqndj\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:04.652046 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.652010    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t" event={"ID":"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00","Type":"ContainerStarted","Data":"dfb6f684f1f8233d389c0a8cb08c0b43a4ef51fc7510215bee309b51a0b0051a"}
Nov 02 23:04:04.680026 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.653332    4294 generic.go:356] "Generic (PLEG): container finished" podID="7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a" containerID="7a20f5c8daacda2fdbc7f96edf79076238c99f9d6e0532381af0538f64689a15" exitCode=0
Nov 02 23:04:04.680026 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.653385    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn" event={"ID":"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a","Type":"ContainerDied","Data":"7a20f5c8daacda2fdbc7f96edf79076238c99f9d6e0532381af0538f64689a15"}
Nov 02 23:04:04.680026 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.653422    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn" event={"ID":"7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a","Type":"ContainerDied","Data":"1b2b40aa426ac6f13ddf2043ce606510dc1dc767dee3b1be0c62a24f785a3373"}
Nov 02 23:04:04.680026 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.653450    4294 scope.go:122] "RemoveContainer" containerID="7a20f5c8daacda2fdbc7f96edf79076238c99f9d6e0532381af0538f64689a15"
Nov 02 23:04:04.680026 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.653595    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"
Nov 02 23:04:04.687743 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.687715    4294 scope.go:122] "RemoveContainer" containerID="7a20f5c8daacda2fdbc7f96edf79076238c99f9d6e0532381af0538f64689a15"
Nov 02 23:04:04.688084 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:04:04.688060    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a20f5c8daacda2fdbc7f96edf79076238c99f9d6e0532381af0538f64689a15\": container with ID starting with 7a20f5c8daacda2fdbc7f96edf79076238c99f9d6e0532381af0538f64689a15 not found: ID does not exist" containerID="7a20f5c8daacda2fdbc7f96edf79076238c99f9d6e0532381af0538f64689a15"
Nov 02 23:04:04.688148 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.688092    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a20f5c8daacda2fdbc7f96edf79076238c99f9d6e0532381af0538f64689a15"} err="failed to get container status \"7a20f5c8daacda2fdbc7f96edf79076238c99f9d6e0532381af0538f64689a15\": rpc error: code = NotFound desc = could not find container \"7a20f5c8daacda2fdbc7f96edf79076238c99f9d6e0532381af0538f64689a15\": container with ID starting with 7a20f5c8daacda2fdbc7f96edf79076238c99f9d6e0532381af0538f64689a15 not found: ID does not exist"
Nov 02 23:04:04.890627 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.890471    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"]
Nov 02 23:04:04.893397 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:04.893372    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0078993-f5f7d5b9b-zxvhn"]
Nov 02 23:04:05.446598 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.446539    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:04:05.507456 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.507419    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:04:05.507785 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.507758    4294 patch_prober.go:29] interesting pod/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 02 23:04:05.507885 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.507813    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t" podUID="57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 02 23:04:05.520751 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.520705    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:05.530313 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.530291    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-token\" (UniqueName: \"kubernetes.io/projected/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-token\") pod \"ea748a41-7475-4fae-a20e-261cd3e210c7\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") "
Nov 02 23:04:05.530398 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.530383    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-data\") pod \"ea748a41-7475-4fae-a20e-261cd3e210c7\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") "
Nov 02 23:04:05.530441 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.530413    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-envoy\") pod \"ea748a41-7475-4fae-a20e-261cd3e210c7\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") "
Nov 02 23:04:05.530525 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.530446    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea748a41-7475-4fae-a20e-261cd3e210c7-kube-api-access-lmxdz\" (UniqueName: \"kubernetes.io/projected/ea748a41-7475-4fae-a20e-261cd3e210c7-kube-api-access-lmxdz\") pod \"ea748a41-7475-4fae-a20e-261cd3e210c7\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") "
Nov 02 23:04:05.530525 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.530468    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-credential-socket\") pod \"ea748a41-7475-4fae-a20e-261cd3e210c7\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") "
Nov 02 23:04:05.530525 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.530489    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-podinfo\") pod \"ea748a41-7475-4fae-a20e-261cd3e210c7\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") "
Nov 02 23:04:05.531233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.530557    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-workload-certs\") pod \"ea748a41-7475-4fae-a20e-261cd3e210c7\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") "
Nov 02 23:04:05.531233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.530591    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea748a41-7475-4fae-a20e-261cd3e210c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea748a41-7475-4fae-a20e-261cd3e210c7-istiod-ca-cert\") pod \"ea748a41-7475-4fae-a20e-261cd3e210c7\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") "
Nov 02 23:04:05.531233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.530638    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-workload-socket\") pod \"ea748a41-7475-4fae-a20e-261cd3e210c7\" (UID: \"ea748a41-7475-4fae-a20e-261cd3e210c7\") "
Nov 02 23:04:05.531233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.530747    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-data" pod "ea748a41-7475-4fae-a20e-261cd3e210c7" (UID: "ea748a41-7475-4fae-a20e-261cd3e210c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:05.531233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.530763    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-credential-socket" pod "ea748a41-7475-4fae-a20e-261cd3e210c7" (UID: "ea748a41-7475-4fae-a20e-261cd3e210c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:05.531233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.530892    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-workload-certs" pod "ea748a41-7475-4fae-a20e-261cd3e210c7" (UID: "ea748a41-7475-4fae-a20e-261cd3e210c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:05.531233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.531047    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-workload-socket" pod "ea748a41-7475-4fae-a20e-261cd3e210c7" (UID: "ea748a41-7475-4fae-a20e-261cd3e210c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:05.531233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.531104    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea748a41-7475-4fae-a20e-261cd3e210c7-istiod-ca-cert" pod "ea748a41-7475-4fae-a20e-261cd3e210c7" (UID: "ea748a41-7475-4fae-a20e-261cd3e210c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:04:05.531233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.531143    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:05.531233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.531158    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:05.531233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.531167    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:05.531233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.531176    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:05.533208 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.533180    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-podinfo" pod "ea748a41-7475-4fae-a20e-261cd3e210c7" (UID: "ea748a41-7475-4fae-a20e-261cd3e210c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:04:05.533295 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.533210    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-envoy" pod "ea748a41-7475-4fae-a20e-261cd3e210c7" (UID: "ea748a41-7475-4fae-a20e-261cd3e210c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:05.533295 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.533224    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea748a41-7475-4fae-a20e-261cd3e210c7-kube-api-access-lmxdz" pod "ea748a41-7475-4fae-a20e-261cd3e210c7" (UID: "ea748a41-7475-4fae-a20e-261cd3e210c7"). InnerVolumeSpecName "kube-api-access-lmxdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:04:05.533295 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.533241    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-token" pod "ea748a41-7475-4fae-a20e-261cd3e210c7" (UID: "ea748a41-7475-4fae-a20e-261cd3e210c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:04:05.632568 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.632530    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:05.632568 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.632566    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmxdz\" (UniqueName: \"kubernetes.io/projected/ea748a41-7475-4fae-a20e-261cd3e210c7-kube-api-access-lmxdz\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:05.632793 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.632582    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:05.632793 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.632595    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea748a41-7475-4fae-a20e-261cd3e210c7-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:05.632793 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.632607    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea748a41-7475-4fae-a20e-261cd3e210c7-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:05.666015 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.665494    4294 generic.go:356] "Generic (PLEG): container finished" podID="ea748a41-7475-4fae-a20e-261cd3e210c7" containerID="1ffbbf6dbf10cd518e0684858c1797ca8684e7da773d2296fa31ff4055909e36" exitCode=0
Nov 02 23:04:05.666015 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.665625    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6" event={"ID":"ea748a41-7475-4fae-a20e-261cd3e210c7","Type":"ContainerDied","Data":"1ffbbf6dbf10cd518e0684858c1797ca8684e7da773d2296fa31ff4055909e36"}
Nov 02 23:04:05.666015 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.665682    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6" event={"ID":"ea748a41-7475-4fae-a20e-261cd3e210c7","Type":"ContainerDied","Data":"b9467c41c049cb9203eede7cb76ac657b98e96a6c2db6850633ff1cee31011dd"}
Nov 02 23:04:05.666015 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.665717    4294 scope.go:122] "RemoveContainer" containerID="1ffbbf6dbf10cd518e0684858c1797ca8684e7da773d2296fa31ff4055909e36"
Nov 02 23:04:05.666015 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.665844    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"
Nov 02 23:04:05.688863 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.688710    4294 scope.go:122] "RemoveContainer" containerID="1ae33ee115630285528f6d1b9fe4a6fb36b01cac1afa20662508993f7106ff76"
Nov 02 23:04:05.697522 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.697502    4294 scope.go:122] "RemoveContainer" containerID="1ffbbf6dbf10cd518e0684858c1797ca8684e7da773d2296fa31ff4055909e36"
Nov 02 23:04:05.697834 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:04:05.697812    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ffbbf6dbf10cd518e0684858c1797ca8684e7da773d2296fa31ff4055909e36\": container with ID starting with 1ffbbf6dbf10cd518e0684858c1797ca8684e7da773d2296fa31ff4055909e36 not found: ID does not exist" containerID="1ffbbf6dbf10cd518e0684858c1797ca8684e7da773d2296fa31ff4055909e36"
Nov 02 23:04:05.697878 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.697845    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffbbf6dbf10cd518e0684858c1797ca8684e7da773d2296fa31ff4055909e36"} err="failed to get container status \"1ffbbf6dbf10cd518e0684858c1797ca8684e7da773d2296fa31ff4055909e36\": rpc error: code = NotFound desc = could not find container \"1ffbbf6dbf10cd518e0684858c1797ca8684e7da773d2296fa31ff4055909e36\": container with ID starting with 1ffbbf6dbf10cd518e0684858c1797ca8684e7da773d2296fa31ff4055909e36 not found: ID does not exist"
Nov 02 23:04:05.697878 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.697867    4294 scope.go:122] "RemoveContainer" containerID="1ae33ee115630285528f6d1b9fe4a6fb36b01cac1afa20662508993f7106ff76"
Nov 02 23:04:05.698148 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:04:05.698129    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ae33ee115630285528f6d1b9fe4a6fb36b01cac1afa20662508993f7106ff76\": container with ID starting with 1ae33ee115630285528f6d1b9fe4a6fb36b01cac1afa20662508993f7106ff76 not found: ID does not exist" containerID="1ae33ee115630285528f6d1b9fe4a6fb36b01cac1afa20662508993f7106ff76"
Nov 02 23:04:05.698202 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.698154    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ae33ee115630285528f6d1b9fe4a6fb36b01cac1afa20662508993f7106ff76"} err="failed to get container status \"1ae33ee115630285528f6d1b9fe4a6fb36b01cac1afa20662508993f7106ff76\": rpc error: code = NotFound desc = could not find container \"1ae33ee115630285528f6d1b9fe4a6fb36b01cac1afa20662508993f7106ff76\": container with ID starting with 1ae33ee115630285528f6d1b9fe4a6fb36b01cac1afa20662508993f7106ff76 not found: ID does not exist"
Nov 02 23:04:05.792922 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.792876    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:05.834258 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.834227    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"]
Nov 02 23:04:05.839096 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.839068    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-8a68c823-gw-openshift-default-66569d674f-h7lx6"]
Nov 02 23:04:05.978721 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:05.978614    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:06.444717 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:06.444676    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:06.508043 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:06.507969    4294 patch_prober.go:29] interesting pod/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 02 23:04:06.508229 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:06.508075    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t" podUID="57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 02 23:04:06.548992 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:06.548941    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a" path="/var/lib/kubelet/pods/7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a/volumes"
Nov 02 23:04:06.549586 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:06.549566    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea748a41-7475-4fae-a20e-261cd3e210c7" path="/var/lib/kubelet/pods/ea748a41-7475-4fae-a20e-261cd3e210c7/volumes"
Nov 02 23:04:06.586035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:06.585995    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"]
Nov 02 23:04:06.586540 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:06.586507    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd" podUID="a3228817-8695-4cea-908c-57cf4bb209bb" containerName="istio-proxy" containerID="cri-o://088ddad49b172503a75cf5ea738a3d5fcff0c07857cc7ad4c2cc207a46057612" gracePeriod=30
Nov 02 23:04:06.631853 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:06.631816    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:06.812175 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:06.812140    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:07.379538 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:07.379505    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:07.507920 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:07.507882    4294 patch_prober.go:29] interesting pod/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 02 23:04:07.508124 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:07.507952    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t" podUID="57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 02 23:04:07.979000 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:07.978949    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:08.227774 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:08.227740    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:08.507744 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:08.507706    4294 patch_prober.go:29] interesting pod/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 02 23:04:08.507903 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:08.507764    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t" podUID="57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 02 23:04:08.779932 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:08.779829    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:08.938370 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:08.938339    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:09.507231 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:09.507185    4294 patch_prober.go:29] interesting pod/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 02 23:04:09.507747 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:09.507284    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t" podUID="57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 02 23:04:09.951106 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:09.951003    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"]
Nov 02 23:04:09.951305 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:09.951246    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q" podUID="d9cef540-e743-42c9-8904-181c1d1c0695" containerName="maas-api" containerID="cri-o://0e9260370b1068cc3de7483e812257d5e98ff2b1aaee514f36e0f463034b644a" gracePeriod=30
Nov 02 23:04:10.122546 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.122515    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:10.224457 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.224386    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"
Nov 02 23:04:10.318517 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.318479    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9cef540-e743-42c9-8904-181c1d1c0695-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9cef540-e743-42c9-8904-181c1d1c0695-openshift-service-ca\") pod \"d9cef540-e743-42c9-8904-181c1d1c0695\" (UID: \"d9cef540-e743-42c9-8904-181c1d1c0695\") "
Nov 02 23:04:10.318690 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.318527    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9cef540-e743-42c9-8904-181c1d1c0695-kube-api-access-fmkjn\" (UniqueName: \"kubernetes.io/projected/d9cef540-e743-42c9-8904-181c1d1c0695-kube-api-access-fmkjn\") pod \"d9cef540-e743-42c9-8904-181c1d1c0695\" (UID: \"d9cef540-e743-42c9-8904-181c1d1c0695\") "
Nov 02 23:04:10.318690 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.318574    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9cef540-e743-42c9-8904-181c1d1c0695-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9cef540-e743-42c9-8904-181c1d1c0695-maas-api-tls\") pod \"d9cef540-e743-42c9-8904-181c1d1c0695\" (UID: \"d9cef540-e743-42c9-8904-181c1d1c0695\") "
Nov 02 23:04:10.318690 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.318642    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9cef540-e743-42c9-8904-181c1d1c0695-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9cef540-e743-42c9-8904-181c1d1c0695-metrics-certs\") pod \"d9cef540-e743-42c9-8904-181c1d1c0695\" (UID: \"d9cef540-e743-42c9-8904-181c1d1c0695\") "
Nov 02 23:04:10.318889 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.318855    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9cef540-e743-42c9-8904-181c1d1c0695-openshift-service-ca" pod "d9cef540-e743-42c9-8904-181c1d1c0695" (UID: "d9cef540-e743-42c9-8904-181c1d1c0695"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:04:10.319104 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.319082    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9cef540-e743-42c9-8904-181c1d1c0695-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:10.320813 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.320785    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9cef540-e743-42c9-8904-181c1d1c0695-kube-api-access-fmkjn" pod "d9cef540-e743-42c9-8904-181c1d1c0695" (UID: "d9cef540-e743-42c9-8904-181c1d1c0695"). InnerVolumeSpecName "kube-api-access-fmkjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:04:10.320911 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.320812    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9cef540-e743-42c9-8904-181c1d1c0695-maas-api-tls" pod "d9cef540-e743-42c9-8904-181c1d1c0695" (UID: "d9cef540-e743-42c9-8904-181c1d1c0695"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:04:10.320911 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.320832    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9cef540-e743-42c9-8904-181c1d1c0695-metrics-certs" pod "d9cef540-e743-42c9-8904-181c1d1c0695" (UID: "d9cef540-e743-42c9-8904-181c1d1c0695"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:04:10.420124 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.420088    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmkjn\" (UniqueName: \"kubernetes.io/projected/d9cef540-e743-42c9-8904-181c1d1c0695-kube-api-access-fmkjn\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:10.420124 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.420117    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9cef540-e743-42c9-8904-181c1d1c0695-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:10.420124 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.420128    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9cef540-e743-42c9-8904-181c1d1c0695-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:10.517657 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.517622    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:04:10.518053 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.517950    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:04:10.518759 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.518741    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:04:10.717390 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.717354    4294 generic.go:356] "Generic (PLEG): container finished" podID="d9cef540-e743-42c9-8904-181c1d1c0695" containerID="0e9260370b1068cc3de7483e812257d5e98ff2b1aaee514f36e0f463034b644a" exitCode=0
Nov 02 23:04:10.717558 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.717411    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"
Nov 02 23:04:10.717558 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.717437    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q" event={"ID":"d9cef540-e743-42c9-8904-181c1d1c0695","Type":"ContainerDied","Data":"0e9260370b1068cc3de7483e812257d5e98ff2b1aaee514f36e0f463034b644a"}
Nov 02 23:04:10.717558 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.717510    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q" event={"ID":"d9cef540-e743-42c9-8904-181c1d1c0695","Type":"ContainerDied","Data":"d0d87e21e89d28ee0f369f639481a49a0571506cc1db5a4bf54dc7719e104210"}
Nov 02 23:04:10.717558 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.717532    4294 scope.go:122] "RemoveContainer" containerID="0e9260370b1068cc3de7483e812257d5e98ff2b1aaee514f36e0f463034b644a"
Nov 02 23:04:10.727355 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.727336    4294 scope.go:122] "RemoveContainer" containerID="0e9260370b1068cc3de7483e812257d5e98ff2b1aaee514f36e0f463034b644a"
Nov 02 23:04:10.727655 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:04:10.727635    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e9260370b1068cc3de7483e812257d5e98ff2b1aaee514f36e0f463034b644a\": container with ID starting with 0e9260370b1068cc3de7483e812257d5e98ff2b1aaee514f36e0f463034b644a not found: ID does not exist" containerID="0e9260370b1068cc3de7483e812257d5e98ff2b1aaee514f36e0f463034b644a"
Nov 02 23:04:10.727697 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.727665    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e9260370b1068cc3de7483e812257d5e98ff2b1aaee514f36e0f463034b644a"} err="failed to get container status \"0e9260370b1068cc3de7483e812257d5e98ff2b1aaee514f36e0f463034b644a\": rpc error: code = NotFound desc = could not find container \"0e9260370b1068cc3de7483e812257d5e98ff2b1aaee514f36e0f463034b644a\": container with ID starting with 0e9260370b1068cc3de7483e812257d5e98ff2b1aaee514f36e0f463034b644a not found: ID does not exist"
Nov 02 23:04:10.853912 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.853818    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"]
Nov 02 23:04:10.860371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:10.860339    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-16aef2f3-794b9cb6dd-fb24q"]
Nov 02 23:04:11.726427 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:11.726392    4294 generic.go:356] "Generic (PLEG): container finished" podID="a3228817-8695-4cea-908c-57cf4bb209bb" containerID="088ddad49b172503a75cf5ea738a3d5fcff0c07857cc7ad4c2cc207a46057612" exitCode=0
Nov 02 23:04:11.726920 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:11.726467    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd" event={"ID":"a3228817-8695-4cea-908c-57cf4bb209bb","Type":"ContainerDied","Data":"088ddad49b172503a75cf5ea738a3d5fcff0c07857cc7ad4c2cc207a46057612"}
Nov 02 23:04:11.726920 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:11.726541    4294 scope.go:122] "RemoveContainer" containerID="3a5664377b50ad21bc46e7bbba5468ce85840d3593c20a873746c5015df9c77f"
Nov 02 23:04:11.750263 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:04:11.750229    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:04:11.859178 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:11.859119    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:04:11.988501 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:11.988445    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:12.037613 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.037578    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-credential-socket\") pod \"a3228817-8695-4cea-908c-57cf4bb209bb\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") "
Nov 02 23:04:12.037770 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.037627    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-istio-envoy\") pod \"a3228817-8695-4cea-908c-57cf4bb209bb\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") "
Nov 02 23:04:12.037770 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.037657    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3228817-8695-4cea-908c-57cf4bb209bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3228817-8695-4cea-908c-57cf4bb209bb-istiod-ca-cert\") pod \"a3228817-8695-4cea-908c-57cf4bb209bb\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") "
Nov 02 23:04:12.037770 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.037693    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-workload-certs\") pod \"a3228817-8695-4cea-908c-57cf4bb209bb\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") "
Nov 02 23:04:12.037770 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.037747    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3228817-8695-4cea-908c-57cf4bb209bb-kube-api-access-t627n\" (UniqueName: \"kubernetes.io/projected/a3228817-8695-4cea-908c-57cf4bb209bb-kube-api-access-t627n\") pod \"a3228817-8695-4cea-908c-57cf4bb209bb\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") "
Nov 02 23:04:12.038008 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.037779    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-workload-socket\") pod \"a3228817-8695-4cea-908c-57cf4bb209bb\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") "
Nov 02 23:04:12.038008 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.037848    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-istio-data\") pod \"a3228817-8695-4cea-908c-57cf4bb209bb\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") "
Nov 02 23:04:12.038008 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.037900    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3228817-8695-4cea-908c-57cf4bb209bb-istio-token\" (UniqueName: \"kubernetes.io/projected/a3228817-8695-4cea-908c-57cf4bb209bb-istio-token\") pod \"a3228817-8695-4cea-908c-57cf4bb209bb\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") "
Nov 02 23:04:12.038008 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.037904    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-credential-socket" pod "a3228817-8695-4cea-908c-57cf4bb209bb" (UID: "a3228817-8695-4cea-908c-57cf4bb209bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:12.038008 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.037930    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3228817-8695-4cea-908c-57cf4bb209bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3228817-8695-4cea-908c-57cf4bb209bb-istio-podinfo\") pod \"a3228817-8695-4cea-908c-57cf4bb209bb\" (UID: \"a3228817-8695-4cea-908c-57cf4bb209bb\") "
Nov 02 23:04:12.038252 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.038040    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-workload-certs" pod "a3228817-8695-4cea-908c-57cf4bb209bb" (UID: "a3228817-8695-4cea-908c-57cf4bb209bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:12.038252 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.038150    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3228817-8695-4cea-908c-57cf4bb209bb-istiod-ca-cert" pod "a3228817-8695-4cea-908c-57cf4bb209bb" (UID: "a3228817-8695-4cea-908c-57cf4bb209bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:04:12.038252 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.038150    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-istio-data" pod "a3228817-8695-4cea-908c-57cf4bb209bb" (UID: "a3228817-8695-4cea-908c-57cf4bb209bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:12.038402 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.038263    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-workload-socket" pod "a3228817-8695-4cea-908c-57cf4bb209bb" (UID: "a3228817-8695-4cea-908c-57cf4bb209bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:12.038491 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.038473    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:12.038553 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.038493    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:12.038553 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.038502    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:12.038553 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.038511    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3228817-8695-4cea-908c-57cf4bb209bb-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:12.038553 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.038523    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:12.040204 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.040177    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3228817-8695-4cea-908c-57cf4bb209bb-istio-token" pod "a3228817-8695-4cea-908c-57cf4bb209bb" (UID: "a3228817-8695-4cea-908c-57cf4bb209bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:04:12.040331 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.040311    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3228817-8695-4cea-908c-57cf4bb209bb-kube-api-access-t627n" pod "a3228817-8695-4cea-908c-57cf4bb209bb" (UID: "a3228817-8695-4cea-908c-57cf4bb209bb"). InnerVolumeSpecName "kube-api-access-t627n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:04:12.040461 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.040442    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-istio-envoy" pod "a3228817-8695-4cea-908c-57cf4bb209bb" (UID: "a3228817-8695-4cea-908c-57cf4bb209bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:12.040496 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.040458    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3228817-8695-4cea-908c-57cf4bb209bb-istio-podinfo" pod "a3228817-8695-4cea-908c-57cf4bb209bb" (UID: "a3228817-8695-4cea-908c-57cf4bb209bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:04:12.139752 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.139662    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3228817-8695-4cea-908c-57cf4bb209bb-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:12.139752 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.139692    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t627n\" (UniqueName: \"kubernetes.io/projected/a3228817-8695-4cea-908c-57cf4bb209bb-kube-api-access-t627n\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:12.139752 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.139704    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3228817-8695-4cea-908c-57cf4bb209bb-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:12.139752 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.139713    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3228817-8695-4cea-908c-57cf4bb209bb-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:12.163213 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.163160    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:12.319228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.319191    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:12.457515 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.457438    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j"]
Nov 02 23:04:12.457854 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.457842    4294 cpu_manager.go:414] "containerMap: removing container" podUID="ea748a41-7475-4fae-a20e-261cd3e210c7" containerName="istio-proxy"
Nov 02 23:04:12.457892 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.457856    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea748a41-7475-4fae-a20e-261cd3e210c7" containerName="istio-proxy"
Nov 02 23:04:12.457892 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.457870    4294 cpu_manager.go:414] "containerMap: removing container" podUID="a3228817-8695-4cea-908c-57cf4bb209bb" containerName="istio-proxy"
Nov 02 23:04:12.457892 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.457876    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3228817-8695-4cea-908c-57cf4bb209bb" containerName="istio-proxy"
Nov 02 23:04:12.457892 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.457887    4294 cpu_manager.go:414] "containerMap: removing container" podUID="d9cef540-e743-42c9-8904-181c1d1c0695" containerName="maas-api"
Nov 02 23:04:12.457892 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.457892    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9cef540-e743-42c9-8904-181c1d1c0695" containerName="maas-api"
Nov 02 23:04:12.458112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.457904    4294 cpu_manager.go:414] "containerMap: removing container" podUID="7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a" containerName="maas-api"
Nov 02 23:04:12.458112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.457909    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a" containerName="maas-api"
Nov 02 23:04:12.458112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.457919    4294 cpu_manager.go:414] "containerMap: removing container" podUID="a3228817-8695-4cea-908c-57cf4bb209bb" containerName="istio-proxy"
Nov 02 23:04:12.458112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.457924    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3228817-8695-4cea-908c-57cf4bb209bb" containerName="istio-proxy"
Nov 02 23:04:12.458112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.457931    4294 cpu_manager.go:414] "containerMap: removing container" podUID="ea748a41-7475-4fae-a20e-261cd3e210c7" containerName="istio-proxy"
Nov 02 23:04:12.458112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.457935    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea748a41-7475-4fae-a20e-261cd3e210c7" containerName="istio-proxy"
Nov 02 23:04:12.458112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.458033    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea748a41-7475-4fae-a20e-261cd3e210c7" containerName="istio-proxy"
Nov 02 23:04:12.458112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.458042    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3228817-8695-4cea-908c-57cf4bb209bb" containerName="istio-proxy"
Nov 02 23:04:12.458112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.458048    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9cef540-e743-42c9-8904-181c1d1c0695" containerName="maas-api"
Nov 02 23:04:12.458112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.458055    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a5cba77-bc09-4f0e-8aed-d301b1b1dd6a" containerName="maas-api"
Nov 02 23:04:12.458112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.458062    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea748a41-7475-4fae-a20e-261cd3e210c7" containerName="istio-proxy"
Nov 02 23:04:12.460101 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.460083    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j"
Nov 02 23:04:12.538596 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.523359    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-14b321-dockercfg-gh9nh\""
Nov 02 23:04:12.538596 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.524059    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-14b321\""
Nov 02 23:04:12.538596 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.526376    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j"]
Nov 02 23:04:12.549277 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.549242    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9cef540-e743-42c9-8904-181c1d1c0695" path="/var/lib/kubelet/pods/d9cef540-e743-42c9-8904-181c1d1c0695/volumes"
Nov 02 23:04:12.635304 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.635256    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k"]
Nov 02 23:04:12.635838 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.635819    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3228817-8695-4cea-908c-57cf4bb209bb" containerName="istio-proxy"
Nov 02 23:04:12.638251 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.638231    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k"
Nov 02 23:04:12.645030 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.645005    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705e82db-2105-4a14-8990-5f0b1322da1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j\" (UID: \"705e82db-2105-4a14-8990-5f0b1322da1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j"
Nov 02 23:04:12.645209 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.645185    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hq79\" (UniqueName: \"kubernetes.io/projected/705e82db-2105-4a14-8990-5f0b1322da1d-kube-api-access-7hq79\") pod \"payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j\" (UID: \"705e82db-2105-4a14-8990-5f0b1322da1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j"
Nov 02 23:04:12.691792 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.691761    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k"]
Nov 02 23:04:12.739121 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.739018    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd" event={"ID":"a3228817-8695-4cea-908c-57cf4bb209bb","Type":"ContainerDied","Data":"fc284eb54de4815dc9bb7ea67a283d1819735039ea02c6da436d36999fd681aa"}
Nov 02 23:04:12.739121 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.739066    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"
Nov 02 23:04:12.739121 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.739119    4294 scope.go:122] "RemoveContainer" containerID="088ddad49b172503a75cf5ea738a3d5fcff0c07857cc7ad4c2cc207a46057612"
Nov 02 23:04:12.746426 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.746398    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705e82db-2105-4a14-8990-5f0b1322da1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j\" (UID: \"705e82db-2105-4a14-8990-5f0b1322da1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j"
Nov 02 23:04:12.746580 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.746443    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c5057a9-8986-4bc3-8a01-2866a8aa09ca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k\" (UID: \"3c5057a9-8986-4bc3-8a01-2866a8aa09ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k"
Nov 02 23:04:12.746580 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.746485    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf4vc\" (UniqueName: \"kubernetes.io/projected/3c5057a9-8986-4bc3-8a01-2866a8aa09ca-kube-api-access-wf4vc\") pod \"payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k\" (UID: \"3c5057a9-8986-4bc3-8a01-2866a8aa09ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k"
Nov 02 23:04:12.746702 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.746676    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hq79\" (UniqueName: \"kubernetes.io/projected/705e82db-2105-4a14-8990-5f0b1322da1d-kube-api-access-7hq79\") pod \"payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j\" (UID: \"705e82db-2105-4a14-8990-5f0b1322da1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j"
Nov 02 23:04:12.747360 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.747321    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705e82db-2105-4a14-8990-5f0b1322da1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j\" (UID: \"705e82db-2105-4a14-8990-5f0b1322da1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j"
Nov 02 23:04:12.800850 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.800817    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hq79\" (UniqueName: \"kubernetes.io/projected/705e82db-2105-4a14-8990-5f0b1322da1d-kube-api-access-7hq79\") pod \"payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j\" (UID: \"705e82db-2105-4a14-8990-5f0b1322da1d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j"
Nov 02 23:04:12.848038 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.847972    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c5057a9-8986-4bc3-8a01-2866a8aa09ca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k\" (UID: \"3c5057a9-8986-4bc3-8a01-2866a8aa09ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k"
Nov 02 23:04:12.848038 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.848046    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf4vc\" (UniqueName: \"kubernetes.io/projected/3c5057a9-8986-4bc3-8a01-2866a8aa09ca-kube-api-access-wf4vc\") pod \"payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k\" (UID: \"3c5057a9-8986-4bc3-8a01-2866a8aa09ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k"
Nov 02 23:04:12.848682 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.848659    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c5057a9-8986-4bc3-8a01-2866a8aa09ca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k\" (UID: \"3c5057a9-8986-4bc3-8a01-2866a8aa09ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k"
Nov 02 23:04:12.852572 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.852544    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"]
Nov 02 23:04:12.856564 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.856538    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d0078993-openshift-default-5d99d8d9c-dq7kd"]
Nov 02 23:04:12.903254 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.903224    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf4vc\" (UniqueName: \"kubernetes.io/projected/3c5057a9-8986-4bc3-8a01-2866a8aa09ca-kube-api-access-wf4vc\") pod \"payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k\" (UID: \"3c5057a9-8986-4bc3-8a01-2866a8aa09ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k"
Nov 02 23:04:12.949600 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:12.949569    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k"
Nov 02 23:04:13.071946 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:13.071908    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j"
Nov 02 23:04:13.314719 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:13.312841    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k"]
Nov 02 23:04:13.315104 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:04:13.315073    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5057a9_8986_4bc3_8a01_2866a8aa09ca.slice/crio-163bf0a5391dbb9dda9239ab8ce407a14ff14861636a557147720b1953421832 WatchSource:0}: Error finding container 163bf0a5391dbb9dda9239ab8ce407a14ff14861636a557147720b1953421832: Status 404 returned error can't find the container with id 163bf0a5391dbb9dda9239ab8ce407a14ff14861636a557147720b1953421832
Nov 02 23:04:13.457118 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:13.457082    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j"]
Nov 02 23:04:13.457943 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:04:13.457916    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705e82db_2105_4a14_8990_5f0b1322da1d.slice/crio-ceb27adb42ed1d14528d38dca1b9256054a441604bb588a03281649efb19b3dc WatchSource:0}: Error finding container ceb27adb42ed1d14528d38dca1b9256054a441604bb588a03281649efb19b3dc: Status 404 returned error can't find the container with id ceb27adb42ed1d14528d38dca1b9256054a441604bb588a03281649efb19b3dc
Nov 02 23:04:13.519560 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:13.519529    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:13.679900 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:13.679806    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:13.747636 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:13.747604    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j" event={"ID":"705e82db-2105-4a14-8990-5f0b1322da1d","Type":"ContainerStarted","Data":"88545ebb05732b51cfd2ef00abc9b36f0413bef82c91db081641dc320fa0adf4"}
Nov 02 23:04:13.748130 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:13.747654    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j" event={"ID":"705e82db-2105-4a14-8990-5f0b1322da1d","Type":"ContainerStarted","Data":"ceb27adb42ed1d14528d38dca1b9256054a441604bb588a03281649efb19b3dc"}
Nov 02 23:04:13.748130 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:13.747702    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j"
Nov 02 23:04:13.749284 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:13.749257    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k" event={"ID":"3c5057a9-8986-4bc3-8a01-2866a8aa09ca","Type":"ContainerStarted","Data":"28e1a01703ed8fe960870caafa4e48832446909a6173b40e0f3122a2b13f448a"}
Nov 02 23:04:13.749420 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:13.749299    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k" event={"ID":"3c5057a9-8986-4bc3-8a01-2866a8aa09ca","Type":"ContainerStarted","Data":"163bf0a5391dbb9dda9239ab8ce407a14ff14861636a557147720b1953421832"}
Nov 02 23:04:13.749420 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:13.749325    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k"
Nov 02 23:04:13.856332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:13.856262    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:13.859313 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:13.859271    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j" podStartSLOduration=1.8592580060000001 podStartE2EDuration="1.859258006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:13.857909234 +0000 UTC m=+693.426264020" watchObservedRunningTime="2026-11-02 23:04:13.859258006 +0000 UTC m=+693.427612792"
Nov 02 23:04:13.970366 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:13.970256    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k" podStartSLOduration=1.9702403400000001 podStartE2EDuration="1.97024034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:13.968349927 +0000 UTC m=+693.536704715" watchObservedRunningTime="2026-11-02 23:04:13.97024034 +0000 UTC m=+693.538595132"
Nov 02 23:04:14.038850 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:14.038814    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:14.549146 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:14.549081    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3228817-8695-4cea-908c-57cf4bb209bb" path="/var/lib/kubelet/pods/a3228817-8695-4cea-908c-57cf4bb209bb/volumes"
Nov 02 23:04:16.827672 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:16.827614    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:20.219869 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:20.219836    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:20.381233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:20.381197    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:21.620939 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:21.620899    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:21.775498 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:21.775463    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:21.926046 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:21.925936    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:24.756375 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:24.756341    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:24.758218 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:24.758192    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j"
Nov 02 23:04:24.758336 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:24.758260    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k"
Nov 02 23:04:26.535078 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:26.535047    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:31.007839 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:31.007785    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:31.159099 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:31.159054    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:31.312681 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:31.312609    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:36.059962 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.059929    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"]
Nov 02 23:04:36.060397 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.060145    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6" podUID="6901cec5-f339-4eca-8a07-70f3bdadd6cf" containerName="maas-api" containerID="cri-o://cca1b030578bb0f7dd049e113ff13b97026a60f16dc006ab5c0b26cdcfeaf790" gracePeriod=30
Nov 02 23:04:36.189666 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.189632    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:36.338113 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.338086    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"
Nov 02 23:04:36.490048 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.490011    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6901cec5-f339-4eca-8a07-70f3bdadd6cf-kube-api-access-5lkk2\" (UniqueName: \"kubernetes.io/projected/6901cec5-f339-4eca-8a07-70f3bdadd6cf-kube-api-access-5lkk2\") pod \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\" (UID: \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\") "
Nov 02 23:04:36.490228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.490101    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6901cec5-f339-4eca-8a07-70f3bdadd6cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6901cec5-f339-4eca-8a07-70f3bdadd6cf-metrics-certs\") pod \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\" (UID: \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\") "
Nov 02 23:04:36.490228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.490168    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6901cec5-f339-4eca-8a07-70f3bdadd6cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6901cec5-f339-4eca-8a07-70f3bdadd6cf-maas-api-tls\") pod \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\" (UID: \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\") "
Nov 02 23:04:36.490301 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.490241    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6901cec5-f339-4eca-8a07-70f3bdadd6cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6901cec5-f339-4eca-8a07-70f3bdadd6cf-openshift-service-ca\") pod \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\" (UID: \"6901cec5-f339-4eca-8a07-70f3bdadd6cf\") "
Nov 02 23:04:36.490647 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.490609    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6901cec5-f339-4eca-8a07-70f3bdadd6cf-openshift-service-ca" pod "6901cec5-f339-4eca-8a07-70f3bdadd6cf" (UID: "6901cec5-f339-4eca-8a07-70f3bdadd6cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:04:36.492261 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.492229    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6901cec5-f339-4eca-8a07-70f3bdadd6cf-maas-api-tls" pod "6901cec5-f339-4eca-8a07-70f3bdadd6cf" (UID: "6901cec5-f339-4eca-8a07-70f3bdadd6cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:04:36.492261 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.492252    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6901cec5-f339-4eca-8a07-70f3bdadd6cf-metrics-certs" pod "6901cec5-f339-4eca-8a07-70f3bdadd6cf" (UID: "6901cec5-f339-4eca-8a07-70f3bdadd6cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:04:36.492434 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.492308    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6901cec5-f339-4eca-8a07-70f3bdadd6cf-kube-api-access-5lkk2" pod "6901cec5-f339-4eca-8a07-70f3bdadd6cf" (UID: "6901cec5-f339-4eca-8a07-70f3bdadd6cf"). InnerVolumeSpecName "kube-api-access-5lkk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:04:36.544694 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.544651    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:36.591400 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.591299    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6901cec5-f339-4eca-8a07-70f3bdadd6cf-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:36.591400 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.591334    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lkk2\" (UniqueName: \"kubernetes.io/projected/6901cec5-f339-4eca-8a07-70f3bdadd6cf-kube-api-access-5lkk2\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:36.591400 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.591349    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6901cec5-f339-4eca-8a07-70f3bdadd6cf-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:36.591400 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.591362    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6901cec5-f339-4eca-8a07-70f3bdadd6cf-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:36.826972 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.826941    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:36.871098 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.871008    4294 generic.go:356] "Generic (PLEG): container finished" podID="6901cec5-f339-4eca-8a07-70f3bdadd6cf" containerID="cca1b030578bb0f7dd049e113ff13b97026a60f16dc006ab5c0b26cdcfeaf790" exitCode=0
Nov 02 23:04:36.871098 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.871066    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"
Nov 02 23:04:36.871098 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.871082    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6" event={"ID":"6901cec5-f339-4eca-8a07-70f3bdadd6cf","Type":"ContainerDied","Data":"cca1b030578bb0f7dd049e113ff13b97026a60f16dc006ab5c0b26cdcfeaf790"}
Nov 02 23:04:36.871374 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.871128    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6" event={"ID":"6901cec5-f339-4eca-8a07-70f3bdadd6cf","Type":"ContainerDied","Data":"934f2c2fe0e2a9036ad80b31c3b066cde3041fa0502fb0b652df117272757f26"}
Nov 02 23:04:36.871374 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.871150    4294 scope.go:122] "RemoveContainer" containerID="cca1b030578bb0f7dd049e113ff13b97026a60f16dc006ab5c0b26cdcfeaf790"
Nov 02 23:04:36.880636 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.880618    4294 scope.go:122] "RemoveContainer" containerID="cca1b030578bb0f7dd049e113ff13b97026a60f16dc006ab5c0b26cdcfeaf790"
Nov 02 23:04:36.880975 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:04:36.880956    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cca1b030578bb0f7dd049e113ff13b97026a60f16dc006ab5c0b26cdcfeaf790\": container with ID starting with cca1b030578bb0f7dd049e113ff13b97026a60f16dc006ab5c0b26cdcfeaf790 not found: ID does not exist" containerID="cca1b030578bb0f7dd049e113ff13b97026a60f16dc006ab5c0b26cdcfeaf790"
Nov 02 23:04:36.881047 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:36.881014    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cca1b030578bb0f7dd049e113ff13b97026a60f16dc006ab5c0b26cdcfeaf790"} err="failed to get container status \"cca1b030578bb0f7dd049e113ff13b97026a60f16dc006ab5c0b26cdcfeaf790\": rpc error: code = NotFound desc = could not find container \"cca1b030578bb0f7dd049e113ff13b97026a60f16dc006ab5c0b26cdcfeaf790\": container with ID starting with cca1b030578bb0f7dd049e113ff13b97026a60f16dc006ab5c0b26cdcfeaf790 not found: ID does not exist"
Nov 02 23:04:37.028838 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.028805    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"]
Nov 02 23:04:37.030748 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.030724    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bdaf93e3-59b74fd976-hbsn6"]
Nov 02 23:04:37.076669 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.076638    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"]
Nov 02 23:04:37.077085 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.076837    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t" podUID="57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" containerName="istio-proxy" containerID="cri-o://dfb6f684f1f8233d389c0a8cb08c0b43a4ef51fc7510215bee309b51a0b0051a" gracePeriod=30
Nov 02 23:04:37.137460 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.137370    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:37.163504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.163468    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"]
Nov 02 23:04:37.163687 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.163662    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b" podUID="ddf27287-71e0-48f6-92d0-3cc184d00e2f" containerName="maas-api" containerID="cri-o://21a0932ee8acd93adea8db8a7b575c9aa1b212f436fc1f8911357f3a035a3175" gracePeriod=30
Nov 02 23:04:37.301362 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.301324    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:37.418476 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.418438    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"
Nov 02 23:04:37.501853 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.501818    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddf27287-71e0-48f6-92d0-3cc184d00e2f-kube-api-access-nv8kt\" (UniqueName: \"kubernetes.io/projected/ddf27287-71e0-48f6-92d0-3cc184d00e2f-kube-api-access-nv8kt\") pod \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\" (UID: \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\") "
Nov 02 23:04:37.502064 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.501943    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddf27287-71e0-48f6-92d0-3cc184d00e2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf27287-71e0-48f6-92d0-3cc184d00e2f-metrics-certs\") pod \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\" (UID: \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\") "
Nov 02 23:04:37.502064 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.502052    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddf27287-71e0-48f6-92d0-3cc184d00e2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf27287-71e0-48f6-92d0-3cc184d00e2f-openshift-service-ca\") pod \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\" (UID: \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\") "
Nov 02 23:04:37.502147 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.502127    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddf27287-71e0-48f6-92d0-3cc184d00e2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf27287-71e0-48f6-92d0-3cc184d00e2f-maas-api-tls\") pod \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\" (UID: \"ddf27287-71e0-48f6-92d0-3cc184d00e2f\") "
Nov 02 23:04:37.502442 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.502415    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf27287-71e0-48f6-92d0-3cc184d00e2f-openshift-service-ca" pod "ddf27287-71e0-48f6-92d0-3cc184d00e2f" (UID: "ddf27287-71e0-48f6-92d0-3cc184d00e2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:04:37.504153 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.504130    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf27287-71e0-48f6-92d0-3cc184d00e2f-maas-api-tls" pod "ddf27287-71e0-48f6-92d0-3cc184d00e2f" (UID: "ddf27287-71e0-48f6-92d0-3cc184d00e2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:04:37.504264 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.504205    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf27287-71e0-48f6-92d0-3cc184d00e2f-metrics-certs" pod "ddf27287-71e0-48f6-92d0-3cc184d00e2f" (UID: "ddf27287-71e0-48f6-92d0-3cc184d00e2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:04:37.504264 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.504207    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf27287-71e0-48f6-92d0-3cc184d00e2f-kube-api-access-nv8kt" pod "ddf27287-71e0-48f6-92d0-3cc184d00e2f" (UID: "ddf27287-71e0-48f6-92d0-3cc184d00e2f"). InnerVolumeSpecName "kube-api-access-nv8kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:04:37.603426 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.603390    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf27287-71e0-48f6-92d0-3cc184d00e2f-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:37.603426 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.603423    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf27287-71e0-48f6-92d0-3cc184d00e2f-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:37.603642 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.603436    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf27287-71e0-48f6-92d0-3cc184d00e2f-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:37.603642 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.603448    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv8kt\" (UniqueName: \"kubernetes.io/projected/ddf27287-71e0-48f6-92d0-3cc184d00e2f-kube-api-access-nv8kt\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:37.652150 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.652056    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:37.878528 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.878492    4294 generic.go:356] "Generic (PLEG): container finished" podID="ddf27287-71e0-48f6-92d0-3cc184d00e2f" containerID="21a0932ee8acd93adea8db8a7b575c9aa1b212f436fc1f8911357f3a035a3175" exitCode=0
Nov 02 23:04:37.878740 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.878537    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"
Nov 02 23:04:37.878740 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.878541    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b" event={"ID":"ddf27287-71e0-48f6-92d0-3cc184d00e2f","Type":"ContainerDied","Data":"21a0932ee8acd93adea8db8a7b575c9aa1b212f436fc1f8911357f3a035a3175"}
Nov 02 23:04:37.878740 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.878594    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b" event={"ID":"ddf27287-71e0-48f6-92d0-3cc184d00e2f","Type":"ContainerDied","Data":"61d3a0bae6d9fc0f1e98e46923071e6e3c57021bb02c5ff1ce0fa1d9ca32fb12"}
Nov 02 23:04:37.878740 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.878622    4294 scope.go:122] "RemoveContainer" containerID="21a0932ee8acd93adea8db8a7b575c9aa1b212f436fc1f8911357f3a035a3175"
Nov 02 23:04:37.888713 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.888694    4294 scope.go:122] "RemoveContainer" containerID="21a0932ee8acd93adea8db8a7b575c9aa1b212f436fc1f8911357f3a035a3175"
Nov 02 23:04:37.889065 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:04:37.889042    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a0932ee8acd93adea8db8a7b575c9aa1b212f436fc1f8911357f3a035a3175\": container with ID starting with 21a0932ee8acd93adea8db8a7b575c9aa1b212f436fc1f8911357f3a035a3175 not found: ID does not exist" containerID="21a0932ee8acd93adea8db8a7b575c9aa1b212f436fc1f8911357f3a035a3175"
Nov 02 23:04:37.889143 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.889077    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a0932ee8acd93adea8db8a7b575c9aa1b212f436fc1f8911357f3a035a3175"} err="failed to get container status \"21a0932ee8acd93adea8db8a7b575c9aa1b212f436fc1f8911357f3a035a3175\": rpc error: code = NotFound desc = could not find container \"21a0932ee8acd93adea8db8a7b575c9aa1b212f436fc1f8911357f3a035a3175\": container with ID starting with 21a0932ee8acd93adea8db8a7b575c9aa1b212f436fc1f8911357f3a035a3175 not found: ID does not exist"
Nov 02 23:04:37.912253 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:37.912166    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:38.035753 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.035710    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"]
Nov 02 23:04:38.042230 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.042193    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-db3e5cc4-76d5fbbb7-thq2b"]
Nov 02 23:04:38.066384 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.066346    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:38.194876 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.194786    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"]
Nov 02 23:04:38.195330 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.195107    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh" podUID="e0eade55-f0dd-4879-841b-bb904df6b7d0" containerName="istio-proxy" containerID="cri-o://7608858135a9b5265a8aa0733366fcbeaa5f23607d921d0bcb22b177dfd37206" gracePeriod=30
Nov 02 23:04:38.254738 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.254705    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:38.477649 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.477552    4294 patch_prober.go:29] interesting pod/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 02 23:04:38.477649 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.477618    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh" podUID="e0eade55-f0dd-4879-841b-bb904df6b7d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 23:04:38.546121 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.546088    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6901cec5-f339-4eca-8a07-70f3bdadd6cf" path="/var/lib/kubelet/pods/6901cec5-f339-4eca-8a07-70f3bdadd6cf/volumes"
Nov 02 23:04:38.546517 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.546503    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddf27287-71e0-48f6-92d0-3cc184d00e2f" path="/var/lib/kubelet/pods/ddf27287-71e0-48f6-92d0-3cc184d00e2f/volumes"
Nov 02 23:04:38.674578 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.674538    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"]
Nov 02 23:04:38.674955 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.674943    4294 cpu_manager.go:414] "containerMap: removing container" podUID="6901cec5-f339-4eca-8a07-70f3bdadd6cf" containerName="maas-api"
Nov 02 23:04:38.675021 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.674956    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6901cec5-f339-4eca-8a07-70f3bdadd6cf" containerName="maas-api"
Nov 02 23:04:38.675021 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.674993    4294 cpu_manager.go:414] "containerMap: removing container" podUID="ddf27287-71e0-48f6-92d0-3cc184d00e2f" containerName="maas-api"
Nov 02 23:04:38.675021 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.674999    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddf27287-71e0-48f6-92d0-3cc184d00e2f" containerName="maas-api"
Nov 02 23:04:38.675119 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.675068    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6901cec5-f339-4eca-8a07-70f3bdadd6cf" containerName="maas-api"
Nov 02 23:04:38.675119 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.675079    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddf27287-71e0-48f6-92d0-3cc184d00e2f" containerName="maas-api"
Nov 02 23:04:38.677840 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.677823    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.731840 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.731747    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ba1918-openshift-default-dockercfg-k6bc8\""
Nov 02 23:04:38.736531 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.736495    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"]
Nov 02 23:04:38.787216 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.787175    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:38.816668 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.816626    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.816859 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.816689    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.816917 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.816844    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.816917 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.816888    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.817015 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.816943    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.817056 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.817012    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.817056 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.817041    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8v54\" (UniqueName: \"kubernetes.io/projected/4571d9d7-7554-444c-a4d7-ba0daf1006fc-kube-api-access-m8v54\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.817131 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.817085    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.817173 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.817133    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.918681 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.918638    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.918888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.918694    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.918888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.918722    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.918888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.918739    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8v54\" (UniqueName: \"kubernetes.io/projected/4571d9d7-7554-444c-a4d7-ba0daf1006fc-kube-api-access-m8v54\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.918888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.918764    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.918888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.918805    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.918888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.918879    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.919275 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.918919    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.919275 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.919002    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.919275 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.919185    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.919430 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.919364    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.919788 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.919503    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.919788 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.919778    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.920034 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.920011    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.921282 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.921254    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.921642 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.921622    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.946451 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.946419    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:38.979668 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.979634    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.979902 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.979878    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8v54\" (UniqueName: \"kubernetes.io/projected/4571d9d7-7554-444c-a4d7-ba0daf1006fc-kube-api-access-m8v54\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:38.989654 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:38.989582    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:39.111650 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:39.111603    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:39.407810 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:39.407782    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"]
Nov 02 23:04:39.409010 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:04:39.408965    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4571d9d7_7554_444c_a4d7_ba0daf1006fc.slice/crio-724d1881f6326cc1f707ac513ce245858cfafd6d74b09a0adab0ae470814c0fa WatchSource:0}: Error finding container 724d1881f6326cc1f707ac513ce245858cfafd6d74b09a0adab0ae470814c0fa: Status 404 returned error can't find the container with id 724d1881f6326cc1f707ac513ce245858cfafd6d74b09a0adab0ae470814c0fa
Nov 02 23:04:39.412097 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:39.412061    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:04:39.412209 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:39.412145    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:04:39.412209 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:39.412191    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:04:39.573927 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:39.573892    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:39.891940 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:39.891902    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr" event={"ID":"4571d9d7-7554-444c-a4d7-ba0daf1006fc","Type":"ContainerStarted","Data":"e1ac82ff6b0c8f8219d1c78859909cddb14aefb2fd130246557b889e7dd143a5"}
Nov 02 23:04:39.892152 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:39.891969    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr" event={"ID":"4571d9d7-7554-444c-a4d7-ba0daf1006fc","Type":"ContainerStarted","Data":"724d1881f6326cc1f707ac513ce245858cfafd6d74b09a0adab0ae470814c0fa"}
Nov 02 23:04:39.939749 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:39.939713    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"]
Nov 02 23:04:39.942220 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:39.942204    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:39.990315 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:39.990276    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:39.992329 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:39.992298    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-3d8404-openshift-default-dockercfg-kh456\""
Nov 02 23:04:40.024120 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.024076    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"]
Nov 02 23:04:40.031352 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.031322    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.031624 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.031601    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da283988-fca2-4bde-8256-4589d9536e4b-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.031701 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.031676    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.031755 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.031733    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da283988-fca2-4bde-8256-4589d9536e4b-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.031851 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.031830    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da283988-fca2-4bde-8256-4589d9536e4b-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.031933 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.031914    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.032006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.031974    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.032084 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.032068    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.032122 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.032094    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjvjl\" (UniqueName: \"kubernetes.io/projected/da283988-fca2-4bde-8256-4589d9536e4b-kube-api-access-vjvjl\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.048465 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.048415    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr" podStartSLOduration=2.04840086 podStartE2EDuration="2.04840086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:40.046053886 +0000 UTC m=+719.614408692" watchObservedRunningTime="2026-11-02 23:04:40.04840086 +0000 UTC m=+719.616755647"
Nov 02 23:04:40.055278 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.055243    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:40.132825 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.132786    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da283988-fca2-4bde-8256-4589d9536e4b-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.133035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.132845    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.133035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.132875    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da283988-fca2-4bde-8256-4589d9536e4b-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.133035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.132895    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da283988-fca2-4bde-8256-4589d9536e4b-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.133035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.132936    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.133035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.132959    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.133035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.133019    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.133351 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.133047    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjvjl\" (UniqueName: \"kubernetes.io/projected/da283988-fca2-4bde-8256-4589d9536e4b-kube-api-access-vjvjl\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.133351 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.133117    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.133525 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.133498    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.133704 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.133552    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.133815 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.133712    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.133815 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.133747    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.133933 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.133815    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da283988-fca2-4bde-8256-4589d9536e4b-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.135334 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.135314    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.135556 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.135538    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da283988-fca2-4bde-8256-4589d9536e4b-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.187738 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.187667    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da283988-fca2-4bde-8256-4589d9536e4b-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.187876 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.187793    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjvjl\" (UniqueName: \"kubernetes.io/projected/da283988-fca2-4bde-8256-4589d9536e4b-kube-api-access-vjvjl\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.255522 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.255485    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:40.259240 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.259217    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:40.520251 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.520217    4294 patch_prober.go:29] interesting pod/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 23:04:40.520611 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.520274    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t" podUID="57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 23:04:40.671259 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.671226    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"]
Nov 02 23:04:40.671417 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:04:40.671394    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda283988_fca2_4bde_8256_4589d9536e4b.slice/crio-516ed4d2840070ebed59f2aae66fea0cfcfcab8a7bd8cd50045fd6b5e10edc0e WatchSource:0}: Error finding container 516ed4d2840070ebed59f2aae66fea0cfcfcab8a7bd8cd50045fd6b5e10edc0e: Status 404 returned error can't find the container with id 516ed4d2840070ebed59f2aae66fea0cfcfcab8a7bd8cd50045fd6b5e10edc0e
Nov 02 23:04:40.673799 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.673770    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:04:40.673871 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.673833    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:04:40.673871 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.673862    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:04:40.898900 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.898809    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29" event={"ID":"da283988-fca2-4bde-8256-4589d9536e4b","Type":"ContainerStarted","Data":"d1735b7f2e08e253b049e20bb5e5d05d34d35673bc7c0a8d396e6fc6a9af6c8c"}
Nov 02 23:04:40.898900 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.898870    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29" event={"ID":"da283988-fca2-4bde-8256-4589d9536e4b","Type":"ContainerStarted","Data":"516ed4d2840070ebed59f2aae66fea0cfcfcab8a7bd8cd50045fd6b5e10edc0e"}
Nov 02 23:04:40.996491 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:40.996458    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:41.011350 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:41.011297    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29" podStartSLOduration=2.011284647 podStartE2EDuration="2.011284647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:41.010879192 +0000 UTC m=+720.579233976" watchObservedRunningTime="2026-11-02 23:04:41.011284647 +0000 UTC m=+720.579639433"
Nov 02 23:04:41.132289 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:41.132256    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"]
Nov 02 23:04:41.132551 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:41.132526    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2" podUID="895718e3-44f1-431d-af81-be3ed2f45b51" containerName="istio-proxy" containerID="cri-o://29eee990be6bb8d3d59343c0bec4f7db74f68b62119a4706e66a24c94289abbc" gracePeriod=30
Nov 02 23:04:41.207939 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:41.207856    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:41.256711 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:41.256674    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:41.265019 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:41.262795    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:41.750082 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:04:41.750046    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:04:41.904170 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:41.904122    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:41.904360 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:41.904303    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:41.905290 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:41.905268    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:04:41.905411 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:41.905318    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:04:41.943959 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:41.943925    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:42.331698 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.331673    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:04:42.460344 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.460252    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-podinfo\") pod \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") "
Nov 02 23:04:42.460344 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.460309    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-kube-api-access-vph52\" (UniqueName: \"kubernetes.io/projected/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-kube-api-access-vph52\") pod \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") "
Nov 02 23:04:42.460577 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.460363    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-token\" (UniqueName: \"kubernetes.io/projected/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-token\") pod \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") "
Nov 02 23:04:42.460577 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.460402    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istiod-ca-cert\") pod \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") "
Nov 02 23:04:42.460577 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.460482    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-credential-socket\") pod \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") "
Nov 02 23:04:42.460577 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.460566    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-data\") pod \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") "
Nov 02 23:04:42.460779 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.460621    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-envoy\") pod \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") "
Nov 02 23:04:42.460779 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.460668    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-workload-socket\") pod \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") "
Nov 02 23:04:42.460779 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.460698    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-workload-certs\") pod \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\" (UID: \"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00\") "
Nov 02 23:04:42.460911 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.460869    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-credential-socket" pod "57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" (UID: "57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:42.460911 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.460891    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istiod-ca-cert" pod "57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" (UID: "57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:04:42.461133 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.461111    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-workload-certs" pod "57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" (UID: "57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:42.461285 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.461266    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:42.461349 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.461291    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:42.461349 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.461306    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:42.461349 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.461305    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-workload-socket" pod "57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" (UID: "57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:42.461349 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.461318    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-data" pod "57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" (UID: "57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:42.462739 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.462710    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-token" pod "57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" (UID: "57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:04:42.463081 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.463059    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-kube-api-access-vph52" pod "57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" (UID: "57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00"). InnerVolumeSpecName "kube-api-access-vph52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:04:42.463164 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.463058    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-podinfo" pod "57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" (UID: "57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:04:42.463284 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.463261    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-envoy" pod "57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" (UID: "57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:42.562159 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.562129    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:42.562159 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.562157    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:42.562333 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.562169    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:42.562333 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.562177    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:42.562333 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.562186    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vph52\" (UniqueName: \"kubernetes.io/projected/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-kube-api-access-vph52\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:42.562333 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.562195    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:42.817233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.817196    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"]
Nov 02 23:04:42.817637 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.817624    4294 cpu_manager.go:414] "containerMap: removing container" podUID="57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" containerName="istio-proxy"
Nov 02 23:04:42.817700 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.817638    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" containerName="istio-proxy"
Nov 02 23:04:42.817700 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.817650    4294 cpu_manager.go:414] "containerMap: removing container" podUID="57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" containerName="istio-proxy"
Nov 02 23:04:42.817700 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.817655    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" containerName="istio-proxy"
Nov 02 23:04:42.817791 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.817725    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" containerName="istio-proxy"
Nov 02 23:04:42.817791 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.817733    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" containerName="istio-proxy"
Nov 02 23:04:42.820026 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.820007    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:42.871944 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.871894    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-fb6e1d-openshift-default-dockercfg-5lz5h\""
Nov 02 23:04:42.889403 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.889364    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:42.901271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.901241    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"]
Nov 02 23:04:42.910910 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.910881    4294 generic.go:356] "Generic (PLEG): container finished" podID="57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" containerID="dfb6f684f1f8233d389c0a8cb08c0b43a4ef51fc7510215bee309b51a0b0051a" exitCode=0
Nov 02 23:04:42.911111 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.910941    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"
Nov 02 23:04:42.911111 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.910960    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t" event={"ID":"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00","Type":"ContainerDied","Data":"dfb6f684f1f8233d389c0a8cb08c0b43a4ef51fc7510215bee309b51a0b0051a"}
Nov 02 23:04:42.911111 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.911021    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t" event={"ID":"57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00","Type":"ContainerDied","Data":"37909bca7ee622220a5bcbca5faa22f4ea2ac97967ad668dd1cf55872f8708d9"}
Nov 02 23:04:42.911111 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.911044    4294 scope.go:122] "RemoveContainer" containerID="dfb6f684f1f8233d389c0a8cb08c0b43a4ef51fc7510215bee309b51a0b0051a"
Nov 02 23:04:42.922208 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.922184    4294 scope.go:122] "RemoveContainer" containerID="1a76b6205325838bec44d3e2c6a2c36e06c8b331e83a224c707a472a649478a3"
Nov 02 23:04:42.931183 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.931157    4294 scope.go:122] "RemoveContainer" containerID="dfb6f684f1f8233d389c0a8cb08c0b43a4ef51fc7510215bee309b51a0b0051a"
Nov 02 23:04:42.931561 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:04:42.931527    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb6f684f1f8233d389c0a8cb08c0b43a4ef51fc7510215bee309b51a0b0051a\": container with ID starting with dfb6f684f1f8233d389c0a8cb08c0b43a4ef51fc7510215bee309b51a0b0051a not found: ID does not exist" containerID="dfb6f684f1f8233d389c0a8cb08c0b43a4ef51fc7510215bee309b51a0b0051a"
Nov 02 23:04:42.931650 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.931564    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb6f684f1f8233d389c0a8cb08c0b43a4ef51fc7510215bee309b51a0b0051a"} err="failed to get container status \"dfb6f684f1f8233d389c0a8cb08c0b43a4ef51fc7510215bee309b51a0b0051a\": rpc error: code = NotFound desc = could not find container \"dfb6f684f1f8233d389c0a8cb08c0b43a4ef51fc7510215bee309b51a0b0051a\": container with ID starting with dfb6f684f1f8233d389c0a8cb08c0b43a4ef51fc7510215bee309b51a0b0051a not found: ID does not exist"
Nov 02 23:04:42.931650 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.931584    4294 scope.go:122] "RemoveContainer" containerID="1a76b6205325838bec44d3e2c6a2c36e06c8b331e83a224c707a472a649478a3"
Nov 02 23:04:42.931909 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:04:42.931889    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a76b6205325838bec44d3e2c6a2c36e06c8b331e83a224c707a472a649478a3\": container with ID starting with 1a76b6205325838bec44d3e2c6a2c36e06c8b331e83a224c707a472a649478a3 not found: ID does not exist" containerID="1a76b6205325838bec44d3e2c6a2c36e06c8b331e83a224c707a472a649478a3"
Nov 02 23:04:42.932034 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.931919    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a76b6205325838bec44d3e2c6a2c36e06c8b331e83a224c707a472a649478a3"} err="failed to get container status \"1a76b6205325838bec44d3e2c6a2c36e06c8b331e83a224c707a472a649478a3\": rpc error: code = NotFound desc = could not find container \"1a76b6205325838bec44d3e2c6a2c36e06c8b331e83a224c707a472a649478a3\": container with ID starting with 1a76b6205325838bec44d3e2c6a2c36e06c8b331e83a224c707a472a649478a3 not found: ID does not exist"
Nov 02 23:04:42.965741 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.965619    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:42.965741 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.965692    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:42.966061 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.965830    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:42.966061 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.965908    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:42.966061 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.966072    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:42.966061 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.966099    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:42.966302 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.966174    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:42.966302 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.966206    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z4bd\" (UniqueName: \"kubernetes.io/projected/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-kube-api-access-4z4bd\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:42.966302 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:42.966237    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:43.045582 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.045543    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"]
Nov 02 23:04:43.047529 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.047505    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-16aef2f3-openshift-default-6dc8547d5c-ltd5t"]
Nov 02 23:04:43.067868 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.067781    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:43.068055 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.067969    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:43.068055 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.068041    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:43.068140 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.068073    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:43.068140 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.068088    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z4bd\" (UniqueName: \"kubernetes.io/projected/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-kube-api-access-4z4bd\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:43.068140 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.068115    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:43.068284 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.068242    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:43.068338 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.068293    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:43.068338 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.068314    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:43.068440 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.068350    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:43.068440 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.068368    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:43.068543 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.068434    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:43.068869 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.068840    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:43.068869 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.068865    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:43.070600 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.070578    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:43.070738 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.070718    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:43.084628 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.084597    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:43.126292 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.126259    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:43.126607 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.126588    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z4bd\" (UniqueName: \"kubernetes.io/projected/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-kube-api-access-4z4bd\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:43.133464 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.133435    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:43.252758 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.252723    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:43.362319 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.362294    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:04:43.471474 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.471435    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0eade55-f0dd-4879-841b-bb904df6b7d0-kube-api-access-2pw6f\" (UniqueName: \"kubernetes.io/projected/e0eade55-f0dd-4879-841b-bb904df6b7d0-kube-api-access-2pw6f\") pod \"e0eade55-f0dd-4879-841b-bb904df6b7d0\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") "
Nov 02 23:04:43.471684 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.471539    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-workload-socket\") pod \"e0eade55-f0dd-4879-841b-bb904df6b7d0\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") "
Nov 02 23:04:43.471684 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.471611    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-workload-certs\") pod \"e0eade55-f0dd-4879-841b-bb904df6b7d0\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") "
Nov 02 23:04:43.471684 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.471673    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-envoy\") pod \"e0eade55-f0dd-4879-841b-bb904df6b7d0\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") "
Nov 02 23:04:43.471847 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.471708    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-token\" (UniqueName: \"kubernetes.io/projected/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-token\") pod \"e0eade55-f0dd-4879-841b-bb904df6b7d0\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") "
Nov 02 23:04:43.471847 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.471766    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0eade55-f0dd-4879-841b-bb904df6b7d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0eade55-f0dd-4879-841b-bb904df6b7d0-istiod-ca-cert\") pod \"e0eade55-f0dd-4879-841b-bb904df6b7d0\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") "
Nov 02 23:04:43.471847 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.471792    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-credential-socket\") pod \"e0eade55-f0dd-4879-841b-bb904df6b7d0\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") "
Nov 02 23:04:43.471847 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.471841    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-podinfo\") pod \"e0eade55-f0dd-4879-841b-bb904df6b7d0\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") "
Nov 02 23:04:43.472086 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.471867    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-data\") pod \"e0eade55-f0dd-4879-841b-bb904df6b7d0\" (UID: \"e0eade55-f0dd-4879-841b-bb904df6b7d0\") "
Nov 02 23:04:43.472086 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.471886    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-workload-certs" pod "e0eade55-f0dd-4879-841b-bb904df6b7d0" (UID: "e0eade55-f0dd-4879-841b-bb904df6b7d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:43.472086 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.471903    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-workload-socket" pod "e0eade55-f0dd-4879-841b-bb904df6b7d0" (UID: "e0eade55-f0dd-4879-841b-bb904df6b7d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:43.472235 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.472119    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-credential-socket" pod "e0eade55-f0dd-4879-841b-bb904df6b7d0" (UID: "e0eade55-f0dd-4879-841b-bb904df6b7d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:43.472235 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.472184    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0eade55-f0dd-4879-841b-bb904df6b7d0-istiod-ca-cert" pod "e0eade55-f0dd-4879-841b-bb904df6b7d0" (UID: "e0eade55-f0dd-4879-841b-bb904df6b7d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:04:43.472363 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.472330    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-data" pod "e0eade55-f0dd-4879-841b-bb904df6b7d0" (UID: "e0eade55-f0dd-4879-841b-bb904df6b7d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:43.472363 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.472353    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:43.472474 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.472374    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0eade55-f0dd-4879-841b-bb904df6b7d0-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:43.472474 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.472386    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:43.472474 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.472395    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:43.473923 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.473902    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-podinfo" pod "e0eade55-f0dd-4879-841b-bb904df6b7d0" (UID: "e0eade55-f0dd-4879-841b-bb904df6b7d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:04:43.474051 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.473941    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0eade55-f0dd-4879-841b-bb904df6b7d0-kube-api-access-2pw6f" pod "e0eade55-f0dd-4879-841b-bb904df6b7d0" (UID: "e0eade55-f0dd-4879-841b-bb904df6b7d0"). InnerVolumeSpecName "kube-api-access-2pw6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:04:43.474245 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.474225    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-envoy" pod "e0eade55-f0dd-4879-841b-bb904df6b7d0" (UID: "e0eade55-f0dd-4879-841b-bb904df6b7d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:43.474292 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.474227    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-token" pod "e0eade55-f0dd-4879-841b-bb904df6b7d0" (UID: "e0eade55-f0dd-4879-841b-bb904df6b7d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:04:43.550882 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.550850    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"]
Nov 02 23:04:43.551659 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:04:43.551630    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d525079_fdb8_4c0e_bb38_2567fcf4a49a.slice/crio-a5f43731456fd3b6e57b65057b5773bc66a92a949f206dac66919824f75c7bb4 WatchSource:0}: Error finding container a5f43731456fd3b6e57b65057b5773bc66a92a949f206dac66919824f75c7bb4: Status 404 returned error can't find the container with id a5f43731456fd3b6e57b65057b5773bc66a92a949f206dac66919824f75c7bb4
Nov 02 23:04:43.554060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.554030    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:04:43.554145 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.554092    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:04:43.554145 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.554120    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:04:43.573523 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.573495    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:43.573523 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.573519    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:43.573692 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.573531    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:43.573692 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.573539    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0eade55-f0dd-4879-841b-bb904df6b7d0-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:43.573692 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.573548    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pw6f\" (UniqueName: \"kubernetes.io/projected/e0eade55-f0dd-4879-841b-bb904df6b7d0-kube-api-access-2pw6f\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:43.918838 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.918730    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp" event={"ID":"9d525079-fdb8-4c0e-bb38-2567fcf4a49a","Type":"ContainerStarted","Data":"58c2c1ee3f93ef3829295a5eee14912143845af9968d63c703eea9ac2c5a9311"}
Nov 02 23:04:43.919354 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.918847    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp" event={"ID":"9d525079-fdb8-4c0e-bb38-2567fcf4a49a","Type":"ContainerStarted","Data":"a5f43731456fd3b6e57b65057b5773bc66a92a949f206dac66919824f75c7bb4"}
Nov 02 23:04:43.928379 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.928340    4294 generic.go:356] "Generic (PLEG): container finished" podID="e0eade55-f0dd-4879-841b-bb904df6b7d0" containerID="7608858135a9b5265a8aa0733366fcbeaa5f23607d921d0bcb22b177dfd37206" exitCode=0
Nov 02 23:04:43.928561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.928408    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"
Nov 02 23:04:43.928561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.928420    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh" event={"ID":"e0eade55-f0dd-4879-841b-bb904df6b7d0","Type":"ContainerDied","Data":"7608858135a9b5265a8aa0733366fcbeaa5f23607d921d0bcb22b177dfd37206"}
Nov 02 23:04:43.928561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.928491    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh" event={"ID":"e0eade55-f0dd-4879-841b-bb904df6b7d0","Type":"ContainerDied","Data":"3a76f1a07b551caad8f77c8a24d5352b627273ee262cdbff3d4b53892bcd74b4"}
Nov 02 23:04:43.928561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.928520    4294 scope.go:122] "RemoveContainer" containerID="7608858135a9b5265a8aa0733366fcbeaa5f23607d921d0bcb22b177dfd37206"
Nov 02 23:04:43.939228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.939194    4294 scope.go:122] "RemoveContainer" containerID="72bb5df9c45760b9238bedc7712c6e64122c29c6eb3cf37116ba6868aad3d8da"
Nov 02 23:04:43.947909 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.947889    4294 scope.go:122] "RemoveContainer" containerID="7608858135a9b5265a8aa0733366fcbeaa5f23607d921d0bcb22b177dfd37206"
Nov 02 23:04:43.948228 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:04:43.948207    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7608858135a9b5265a8aa0733366fcbeaa5f23607d921d0bcb22b177dfd37206\": container with ID starting with 7608858135a9b5265a8aa0733366fcbeaa5f23607d921d0bcb22b177dfd37206 not found: ID does not exist" containerID="7608858135a9b5265a8aa0733366fcbeaa5f23607d921d0bcb22b177dfd37206"
Nov 02 23:04:43.948271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.948239    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7608858135a9b5265a8aa0733366fcbeaa5f23607d921d0bcb22b177dfd37206"} err="failed to get container status \"7608858135a9b5265a8aa0733366fcbeaa5f23607d921d0bcb22b177dfd37206\": rpc error: code = NotFound desc = could not find container \"7608858135a9b5265a8aa0733366fcbeaa5f23607d921d0bcb22b177dfd37206\": container with ID starting with 7608858135a9b5265a8aa0733366fcbeaa5f23607d921d0bcb22b177dfd37206 not found: ID does not exist"
Nov 02 23:04:43.948271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.948259    4294 scope.go:122] "RemoveContainer" containerID="72bb5df9c45760b9238bedc7712c6e64122c29c6eb3cf37116ba6868aad3d8da"
Nov 02 23:04:43.948540 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:04:43.948521    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72bb5df9c45760b9238bedc7712c6e64122c29c6eb3cf37116ba6868aad3d8da\": container with ID starting with 72bb5df9c45760b9238bedc7712c6e64122c29c6eb3cf37116ba6868aad3d8da not found: ID does not exist" containerID="72bb5df9c45760b9238bedc7712c6e64122c29c6eb3cf37116ba6868aad3d8da"
Nov 02 23:04:43.948579 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.948548    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72bb5df9c45760b9238bedc7712c6e64122c29c6eb3cf37116ba6868aad3d8da"} err="failed to get container status \"72bb5df9c45760b9238bedc7712c6e64122c29c6eb3cf37116ba6868aad3d8da\": rpc error: code = NotFound desc = could not find container \"72bb5df9c45760b9238bedc7712c6e64122c29c6eb3cf37116ba6868aad3d8da\": container with ID starting with 72bb5df9c45760b9238bedc7712c6e64122c29c6eb3cf37116ba6868aad3d8da not found: ID does not exist"
Nov 02 23:04:43.952948 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:43.952929    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:44.036912 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:44.036852    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp" podStartSLOduration=2.036837187 podStartE2EDuration="2.036837187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:44.035231681 +0000 UTC m=+723.603586469" watchObservedRunningTime="2026-11-02 23:04:44.036837187 +0000 UTC m=+723.605191973"
Nov 02 23:04:44.133785 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:44.133746    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:44.139390 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:44.139359    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:44.141699 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:44.141677    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"]
Nov 02 23:04:44.145564 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:44.145537    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-bdaf93e3-openshift-default-5fd8bb7565-dd5gh"]
Nov 02 23:04:44.428269 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:44.428236    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:44.547239 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:44.547194    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00" path="/var/lib/kubelet/pods/57a60fe3-d55f-4bc2-95bf-64f6fb2b0d00/volumes"
Nov 02 23:04:44.547766 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:44.547748    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0eade55-f0dd-4879-841b-bb904df6b7d0" path="/var/lib/kubelet/pods/e0eade55-f0dd-4879-841b-bb904df6b7d0/volumes"
Nov 02 23:04:44.603225 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:44.603191    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:44.775199 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:44.775169    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:44.936899 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:44.936866    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:44.937859 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:44.937838    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:04:45.591166 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:45.591127    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:46.170963 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.170824    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:46.338610 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.338576    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:46.424114 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.424090    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:04:46.611274 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.611229    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/895718e3-44f1-431d-af81-be3ed2f45b51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895718e3-44f1-431d-af81-be3ed2f45b51-istio-podinfo\") pod \"895718e3-44f1-431d-af81-be3ed2f45b51\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") "
Nov 02 23:04:46.611274 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.611278    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-credential-socket\") pod \"895718e3-44f1-431d-af81-be3ed2f45b51\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") "
Nov 02 23:04:46.611542 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.611329    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895718e3-44f1-431d-af81-be3ed2f45b51-kube-api-access-jljkx\" (UniqueName: \"kubernetes.io/projected/895718e3-44f1-431d-af81-be3ed2f45b51-kube-api-access-jljkx\") pod \"895718e3-44f1-431d-af81-be3ed2f45b51\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") "
Nov 02 23:04:46.611542 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.611356    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895718e3-44f1-431d-af81-be3ed2f45b51-istio-token\" (UniqueName: \"kubernetes.io/projected/895718e3-44f1-431d-af81-be3ed2f45b51-istio-token\") pod \"895718e3-44f1-431d-af81-be3ed2f45b51\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") "
Nov 02 23:04:46.611542 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.611392    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/895718e3-44f1-431d-af81-be3ed2f45b51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895718e3-44f1-431d-af81-be3ed2f45b51-istiod-ca-cert\") pod \"895718e3-44f1-431d-af81-be3ed2f45b51\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") "
Nov 02 23:04:46.611542 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.611423    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-workload-certs\") pod \"895718e3-44f1-431d-af81-be3ed2f45b51\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") "
Nov 02 23:04:46.611542 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.611516    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-istio-data\") pod \"895718e3-44f1-431d-af81-be3ed2f45b51\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") "
Nov 02 23:04:46.611805 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.611560    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-workload-socket\") pod \"895718e3-44f1-431d-af81-be3ed2f45b51\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") "
Nov 02 23:04:46.611805 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.611594    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-istio-envoy\") pod \"895718e3-44f1-431d-af81-be3ed2f45b51\" (UID: \"895718e3-44f1-431d-af81-be3ed2f45b51\") "
Nov 02 23:04:46.611805 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.611634    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-credential-socket" pod "895718e3-44f1-431d-af81-be3ed2f45b51" (UID: "895718e3-44f1-431d-af81-be3ed2f45b51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:46.611970 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.611835    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-workload-certs" pod "895718e3-44f1-431d-af81-be3ed2f45b51" (UID: "895718e3-44f1-431d-af81-be3ed2f45b51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:46.611970 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.611932    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-workload-socket" pod "895718e3-44f1-431d-af81-be3ed2f45b51" (UID: "895718e3-44f1-431d-af81-be3ed2f45b51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:46.612117 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.612062    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:46.612117 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.612083    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:46.612117 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.612100    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:46.612315 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.612292    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-istio-data" pod "895718e3-44f1-431d-af81-be3ed2f45b51" (UID: "895718e3-44f1-431d-af81-be3ed2f45b51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:46.612412 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.612382    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895718e3-44f1-431d-af81-be3ed2f45b51-istiod-ca-cert" pod "895718e3-44f1-431d-af81-be3ed2f45b51" (UID: "895718e3-44f1-431d-af81-be3ed2f45b51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:04:46.613851 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.613822    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895718e3-44f1-431d-af81-be3ed2f45b51-kube-api-access-jljkx" pod "895718e3-44f1-431d-af81-be3ed2f45b51" (UID: "895718e3-44f1-431d-af81-be3ed2f45b51"). InnerVolumeSpecName "kube-api-access-jljkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:04:46.614011 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.613964    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-istio-envoy" pod "895718e3-44f1-431d-af81-be3ed2f45b51" (UID: "895718e3-44f1-431d-af81-be3ed2f45b51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:04:46.614429 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.614400    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895718e3-44f1-431d-af81-be3ed2f45b51-istio-token" pod "895718e3-44f1-431d-af81-be3ed2f45b51" (UID: "895718e3-44f1-431d-af81-be3ed2f45b51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:04:46.614429 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.614405    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/895718e3-44f1-431d-af81-be3ed2f45b51-istio-podinfo" pod "895718e3-44f1-431d-af81-be3ed2f45b51" (UID: "895718e3-44f1-431d-af81-be3ed2f45b51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:04:46.712783 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.712685    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jljkx\" (UniqueName: \"kubernetes.io/projected/895718e3-44f1-431d-af81-be3ed2f45b51-kube-api-access-jljkx\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:46.712783 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.712720    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895718e3-44f1-431d-af81-be3ed2f45b51-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:46.712783 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.712729    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895718e3-44f1-431d-af81-be3ed2f45b51-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:46.712783 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.712742    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:46.712783 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.712756    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895718e3-44f1-431d-af81-be3ed2f45b51-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:46.712783 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.712766    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895718e3-44f1-431d-af81-be3ed2f45b51-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:04:46.882435 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.882401    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:46.952750 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.952713    4294 generic.go:356] "Generic (PLEG): container finished" podID="895718e3-44f1-431d-af81-be3ed2f45b51" containerID="29eee990be6bb8d3d59343c0bec4f7db74f68b62119a4706e66a24c94289abbc" exitCode=0
Nov 02 23:04:46.952959 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.952772    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"
Nov 02 23:04:46.952959 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.952793    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2" event={"ID":"895718e3-44f1-431d-af81-be3ed2f45b51","Type":"ContainerDied","Data":"29eee990be6bb8d3d59343c0bec4f7db74f68b62119a4706e66a24c94289abbc"}
Nov 02 23:04:46.952959 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.952850    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2" event={"ID":"895718e3-44f1-431d-af81-be3ed2f45b51","Type":"ContainerDied","Data":"ef4e8b7229a63017dc9a83e6f667da74a88384bb4fa6c142587ff07e57ba92df"}
Nov 02 23:04:46.952959 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.952864    4294 scope.go:122] "RemoveContainer" containerID="29eee990be6bb8d3d59343c0bec4f7db74f68b62119a4706e66a24c94289abbc"
Nov 02 23:04:46.965900 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.965870    4294 scope.go:122] "RemoveContainer" containerID="319fe6f3a44a883039fd3be1671068f42e560c107beed00fa71acf838a9c5209"
Nov 02 23:04:46.977372 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.977348    4294 scope.go:122] "RemoveContainer" containerID="29eee990be6bb8d3d59343c0bec4f7db74f68b62119a4706e66a24c94289abbc"
Nov 02 23:04:46.977743 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:04:46.977720    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29eee990be6bb8d3d59343c0bec4f7db74f68b62119a4706e66a24c94289abbc\": container with ID starting with 29eee990be6bb8d3d59343c0bec4f7db74f68b62119a4706e66a24c94289abbc not found: ID does not exist" containerID="29eee990be6bb8d3d59343c0bec4f7db74f68b62119a4706e66a24c94289abbc"
Nov 02 23:04:46.977843 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.977759    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29eee990be6bb8d3d59343c0bec4f7db74f68b62119a4706e66a24c94289abbc"} err="failed to get container status \"29eee990be6bb8d3d59343c0bec4f7db74f68b62119a4706e66a24c94289abbc\": rpc error: code = NotFound desc = could not find container \"29eee990be6bb8d3d59343c0bec4f7db74f68b62119a4706e66a24c94289abbc\": container with ID starting with 29eee990be6bb8d3d59343c0bec4f7db74f68b62119a4706e66a24c94289abbc not found: ID does not exist"
Nov 02 23:04:46.977843 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.977785    4294 scope.go:122] "RemoveContainer" containerID="319fe6f3a44a883039fd3be1671068f42e560c107beed00fa71acf838a9c5209"
Nov 02 23:04:46.978115 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:04:46.978092    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319fe6f3a44a883039fd3be1671068f42e560c107beed00fa71acf838a9c5209\": container with ID starting with 319fe6f3a44a883039fd3be1671068f42e560c107beed00fa71acf838a9c5209 not found: ID does not exist" containerID="319fe6f3a44a883039fd3be1671068f42e560c107beed00fa71acf838a9c5209"
Nov 02 23:04:46.978159 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:46.978122    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319fe6f3a44a883039fd3be1671068f42e560c107beed00fa71acf838a9c5209"} err="failed to get container status \"319fe6f3a44a883039fd3be1671068f42e560c107beed00fa71acf838a9c5209\": rpc error: code = NotFound desc = could not find container \"319fe6f3a44a883039fd3be1671068f42e560c107beed00fa71acf838a9c5209\": container with ID starting with 319fe6f3a44a883039fd3be1671068f42e560c107beed00fa71acf838a9c5209 not found: ID does not exist"
Nov 02 23:04:47.114328 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.114292    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"]
Nov 02 23:04:47.120216 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.120189    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:47.120385 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.120361    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-db3e5cc4-openshift-default-7946878cc5-krwg2"]
Nov 02 23:04:47.258861 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.258827    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj"]
Nov 02 23:04:47.259275 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.259265    4294 cpu_manager.go:414] "containerMap: removing container" podUID="e0eade55-f0dd-4879-841b-bb904df6b7d0" containerName="istio-proxy"
Nov 02 23:04:47.259315 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.259278    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0eade55-f0dd-4879-841b-bb904df6b7d0" containerName="istio-proxy"
Nov 02 23:04:47.259315 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.259294    4294 cpu_manager.go:414] "containerMap: removing container" podUID="e0eade55-f0dd-4879-841b-bb904df6b7d0" containerName="istio-proxy"
Nov 02 23:04:47.259315 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.259299    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0eade55-f0dd-4879-841b-bb904df6b7d0" containerName="istio-proxy"
Nov 02 23:04:47.259315 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.259307    4294 cpu_manager.go:414] "containerMap: removing container" podUID="895718e3-44f1-431d-af81-be3ed2f45b51" containerName="istio-proxy"
Nov 02 23:04:47.259315 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.259312    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895718e3-44f1-431d-af81-be3ed2f45b51" containerName="istio-proxy"
Nov 02 23:04:47.259460 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.259380    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0eade55-f0dd-4879-841b-bb904df6b7d0" containerName="istio-proxy"
Nov 02 23:04:47.259460 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.259389    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0eade55-f0dd-4879-841b-bb904df6b7d0" containerName="istio-proxy"
Nov 02 23:04:47.259460 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.259395    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="895718e3-44f1-431d-af81-be3ed2f45b51" containerName="istio-proxy"
Nov 02 23:04:47.262311 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.262292    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj"
Nov 02 23:04:47.313109 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.313064    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-ba1918\""
Nov 02 23:04:47.315645 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.315618    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj"]
Nov 02 23:04:47.317794 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.317767    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924707d4-bd50-4ce6-9d59-b9abf9d41f64-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj\" (UID: \"924707d4-bd50-4ce6-9d59-b9abf9d41f64\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj"
Nov 02 23:04:47.317951 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.317926    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcjnc\" (UniqueName: \"kubernetes.io/projected/924707d4-bd50-4ce6-9d59-b9abf9d41f64-kube-api-access-bcjnc\") pod \"payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj\" (UID: \"924707d4-bd50-4ce6-9d59-b9abf9d41f64\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj"
Nov 02 23:04:47.328597 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.328571    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-ba1918-dockercfg-mg9zv\""
Nov 02 23:04:47.419104 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.419066    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924707d4-bd50-4ce6-9d59-b9abf9d41f64-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj\" (UID: \"924707d4-bd50-4ce6-9d59-b9abf9d41f64\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj"
Nov 02 23:04:47.419284 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.419258    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcjnc\" (UniqueName: \"kubernetes.io/projected/924707d4-bd50-4ce6-9d59-b9abf9d41f64-kube-api-access-bcjnc\") pod \"payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj\" (UID: \"924707d4-bd50-4ce6-9d59-b9abf9d41f64\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj"
Nov 02 23:04:47.419757 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.419732    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924707d4-bd50-4ce6-9d59-b9abf9d41f64-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj\" (UID: \"924707d4-bd50-4ce6-9d59-b9abf9d41f64\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj"
Nov 02 23:04:47.421662 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.421642    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7"]
Nov 02 23:04:47.422112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.422099    4294 cpu_manager.go:414] "containerMap: removing container" podUID="895718e3-44f1-431d-af81-be3ed2f45b51" containerName="istio-proxy"
Nov 02 23:04:47.422156 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.422113    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895718e3-44f1-431d-af81-be3ed2f45b51" containerName="istio-proxy"
Nov 02 23:04:47.422192 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.422182    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="895718e3-44f1-431d-af81-be3ed2f45b51" containerName="istio-proxy"
Nov 02 23:04:47.429134 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.426312    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7"
Nov 02 23:04:47.473540 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.473506    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcjnc\" (UniqueName: \"kubernetes.io/projected/924707d4-bd50-4ce6-9d59-b9abf9d41f64-kube-api-access-bcjnc\") pod \"payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj\" (UID: \"924707d4-bd50-4ce6-9d59-b9abf9d41f64\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj"
Nov 02 23:04:47.478074 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.478034    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7"]
Nov 02 23:04:47.520468 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.520369    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4drb\" (UniqueName: \"kubernetes.io/projected/959479c1-5aad-4927-9d79-384ef72ebf9e-kube-api-access-x4drb\") pod \"payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7\" (UID: \"959479c1-5aad-4927-9d79-384ef72ebf9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7"
Nov 02 23:04:47.520652 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.520579    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959479c1-5aad-4927-9d79-384ef72ebf9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7\" (UID: \"959479c1-5aad-4927-9d79-384ef72ebf9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7"
Nov 02 23:04:47.574267 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.574231    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj"
Nov 02 23:04:47.621434 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.621380    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959479c1-5aad-4927-9d79-384ef72ebf9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7\" (UID: \"959479c1-5aad-4927-9d79-384ef72ebf9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7"
Nov 02 23:04:47.621585 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.621440    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4drb\" (UniqueName: \"kubernetes.io/projected/959479c1-5aad-4927-9d79-384ef72ebf9e-kube-api-access-x4drb\") pod \"payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7\" (UID: \"959479c1-5aad-4927-9d79-384ef72ebf9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7"
Nov 02 23:04:47.622489 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.622462    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959479c1-5aad-4927-9d79-384ef72ebf9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7\" (UID: \"959479c1-5aad-4927-9d79-384ef72ebf9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7"
Nov 02 23:04:47.676344 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.676302    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4drb\" (UniqueName: \"kubernetes.io/projected/959479c1-5aad-4927-9d79-384ef72ebf9e-kube-api-access-x4drb\") pod \"payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7\" (UID: \"959479c1-5aad-4927-9d79-384ef72ebf9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7"
Nov 02 23:04:47.740867 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.740834    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7"
Nov 02 23:04:47.940968 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.940938    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj"]
Nov 02 23:04:47.942951 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:04:47.942928    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924707d4_bd50_4ce6_9d59_b9abf9d41f64.slice/crio-f2496f4ba02eaa5b4cc07b3e418fc87a7a6fdffe0de6864c2000f64bf515729b WatchSource:0}: Error finding container f2496f4ba02eaa5b4cc07b3e418fc87a7a6fdffe0de6864c2000f64bf515729b: Status 404 returned error can't find the container with id f2496f4ba02eaa5b4cc07b3e418fc87a7a6fdffe0de6864c2000f64bf515729b
Nov 02 23:04:47.959337 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:47.959305    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj" event={"ID":"924707d4-bd50-4ce6-9d59-b9abf9d41f64","Type":"ContainerStarted","Data":"f2496f4ba02eaa5b4cc07b3e418fc87a7a6fdffe0de6864c2000f64bf515729b"}
Nov 02 23:04:48.108749 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:48.108722    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7"]
Nov 02 23:04:48.109449 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:04:48.109425    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959479c1_5aad_4927_9d79_384ef72ebf9e.slice/crio-47ac25150737d85c25137456ce4a7910192ea454f3e99823e0c6b8f5b9373885 WatchSource:0}: Error finding container 47ac25150737d85c25137456ce4a7910192ea454f3e99823e0c6b8f5b9373885: Status 404 returned error can't find the container with id 47ac25150737d85c25137456ce4a7910192ea454f3e99823e0c6b8f5b9373885
Nov 02 23:04:48.548539 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:48.548496    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="895718e3-44f1-431d-af81-be3ed2f45b51" path="/var/lib/kubelet/pods/895718e3-44f1-431d-af81-be3ed2f45b51/volumes"
Nov 02 23:04:48.886701 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:48.886611    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:48.966663 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:48.966616    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj" event={"ID":"924707d4-bd50-4ce6-9d59-b9abf9d41f64","Type":"ContainerStarted","Data":"12aadbb0dc40ddd4e4ec2d9c212d1d7b8cb08a304c38c7735a4b2ab95ed7e8f4"}
Nov 02 23:04:48.966942 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:48.966687    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj"
Nov 02 23:04:48.968952 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:48.968922    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7" event={"ID":"959479c1-5aad-4927-9d79-384ef72ebf9e","Type":"ContainerStarted","Data":"9d6bb2e39dd7b760a297d506b011c6395b8fe234a087751b4eb88f7bd21c3b49"}
Nov 02 23:04:48.969092 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:48.969015    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7"
Nov 02 23:04:48.969092 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:48.969044    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7" event={"ID":"959479c1-5aad-4927-9d79-384ef72ebf9e","Type":"ContainerStarted","Data":"47ac25150737d85c25137456ce4a7910192ea454f3e99823e0c6b8f5b9373885"}
Nov 02 23:04:49.065034 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:49.064964    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:49.075964 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:49.075911    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj" podStartSLOduration=2.075898067 podStartE2EDuration="2.075898067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:49.075561205 +0000 UTC m=+728.643915993" watchObservedRunningTime="2026-11-02 23:04:49.075898067 +0000 UTC m=+728.644252854"
Nov 02 23:04:49.191471 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:49.191346    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7" podStartSLOduration=2.191326111 podStartE2EDuration="2.191326111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:49.186768522 +0000 UTC m=+728.755123310" watchObservedRunningTime="2026-11-02 23:04:49.191326111 +0000 UTC m=+728.759680897"
Nov 02 23:04:49.236881 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:49.236845    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:49.933277 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:49.933241    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"]
Nov 02 23:04:49.935861 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:49.935842    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:49.945064 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:49.945039    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-model-cache\") pod \"simulator-w3-14b321-kserve-5758674467-z72pg\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") " pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:49.945184 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:49.945096    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-home\") pod \"simulator-w3-14b321-kserve-5758674467-z72pg\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") " pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:49.945227 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:49.945193    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-dshm\") pod \"simulator-w3-14b321-kserve-5758674467-z72pg\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") " pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:49.945266 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:49.945224    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/845035ca-c55e-4e32-a19a-dc3617e3d638-tls-certs\") pod \"simulator-w3-14b321-kserve-5758674467-z72pg\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") " pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:49.945359 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:49.945338    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt7rp\" (UniqueName: \"kubernetes.io/projected/845035ca-c55e-4e32-a19a-dc3617e3d638-kube-api-access-mt7rp\") pod \"simulator-w3-14b321-kserve-5758674467-z72pg\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") " pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:49.945402 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:49.945393    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-tmp-dir\") pod \"simulator-w3-14b321-kserve-5758674467-z72pg\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") " pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:49.991183 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:49.991143    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-14b321\"/\"openshift-service-ca.crt\""
Nov 02 23:04:49.993012 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:49.991420    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-14b321\"/\"default-dockercfg-px9j7\""
Nov 02 23:04:49.993012 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:49.991512    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-14b321\"/\"kube-root-ca.crt\""
Nov 02 23:04:49.993012 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:49.991591    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-14b321\"/\"simulator-w3-14b321-kserve-self-signed-certs\""
Nov 02 23:04:49.993818 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:49.993795    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"]
Nov 02 23:04:50.046430 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:50.046388    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-tmp-dir\") pod \"simulator-w3-14b321-kserve-5758674467-z72pg\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") " pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:50.046633 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:50.046482    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-model-cache\") pod \"simulator-w3-14b321-kserve-5758674467-z72pg\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") " pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:50.046633 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:50.046531    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-home\") pod \"simulator-w3-14b321-kserve-5758674467-z72pg\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") " pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:50.046750 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:50.046705    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-dshm\") pod \"simulator-w3-14b321-kserve-5758674467-z72pg\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") " pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:50.046815 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:50.046747    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/845035ca-c55e-4e32-a19a-dc3617e3d638-tls-certs\") pod \"simulator-w3-14b321-kserve-5758674467-z72pg\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") " pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:50.046879 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:50.046845    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-tmp-dir\") pod \"simulator-w3-14b321-kserve-5758674467-z72pg\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") " pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:50.046929 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:50.046902    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-home\") pod \"simulator-w3-14b321-kserve-5758674467-z72pg\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") " pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:50.046970 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:50.046953    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt7rp\" (UniqueName: \"kubernetes.io/projected/845035ca-c55e-4e32-a19a-dc3617e3d638-kube-api-access-mt7rp\") pod \"simulator-w3-14b321-kserve-5758674467-z72pg\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") " pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:50.047151 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:50.047124    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-model-cache\") pod \"simulator-w3-14b321-kserve-5758674467-z72pg\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") " pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:50.049019 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:50.048997    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-dshm\") pod \"simulator-w3-14b321-kserve-5758674467-z72pg\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") " pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:50.049326 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:50.049307    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/845035ca-c55e-4e32-a19a-dc3617e3d638-tls-certs\") pod \"simulator-w3-14b321-kserve-5758674467-z72pg\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") " pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:50.056379 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:50.056351    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:50.101809 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:50.101779    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt7rp\" (UniqueName: \"kubernetes.io/projected/845035ca-c55e-4e32-a19a-dc3617e3d638-kube-api-access-mt7rp\") pod \"simulator-w3-14b321-kserve-5758674467-z72pg\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") " pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:50.246923 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:50.246891    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:50.612058 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:50.612032    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"]
Nov 02 23:04:50.613938 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:04:50.613906    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod845035ca_c55e_4e32_a19a_dc3617e3d638.slice/crio-79bf2253e56b7ab77716fca065c511f430145da6b6577f50255af1720f0d4da7 WatchSource:0}: Error finding container 79bf2253e56b7ab77716fca065c511f430145da6b6577f50255af1720f0d4da7: Status 404 returned error can't find the container with id 79bf2253e56b7ab77716fca065c511f430145da6b6577f50255af1720f0d4da7
Nov 02 23:04:50.981753 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:50.981666    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" event={"ID":"845035ca-c55e-4e32-a19a-dc3617e3d638","Type":"ContainerStarted","Data":"281569a0a2b76c0af4dfc908efb83b1f70c09f0f5ba313c3d0b82af2f638222f"}
Nov 02 23:04:50.981753 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:50.981710    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" event={"ID":"845035ca-c55e-4e32-a19a-dc3617e3d638","Type":"ContainerStarted","Data":"79bf2253e56b7ab77716fca065c511f430145da6b6577f50255af1720f0d4da7"}
Nov 02 23:04:51.091673 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:51.091617    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" podStartSLOduration=2.09160204 podStartE2EDuration="2.09160204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:51.090634734 +0000 UTC m=+730.658989532" watchObservedRunningTime="2026-11-02 23:04:51.09160204 +0000 UTC m=+730.659956827"
Nov 02 23:04:51.247483 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:51.247452    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:04:51.456049 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:51.456006    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:51.622061 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:51.621938    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:51.829305 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:51.829268    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f"]
Nov 02 23:04:51.831853 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:51.831826    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f"
Nov 02 23:04:51.863520 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:51.863485    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7m7p\" (UniqueName: \"kubernetes.io/projected/b716a68d-1d7e-48d6-98a5-6fdc5472cac9-kube-api-access-k7m7p\") pod \"payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f\" (UID: \"b716a68d-1d7e-48d6-98a5-6fdc5472cac9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f"
Nov 02 23:04:51.863726 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:51.863554    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b716a68d-1d7e-48d6-98a5-6fdc5472cac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f\" (UID: \"b716a68d-1d7e-48d6-98a5-6fdc5472cac9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f"
Nov 02 23:04:51.888825 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:51.885972    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-3d8404\""
Nov 02 23:04:51.888825 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:51.888706    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f"]
Nov 02 23:04:51.932599 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:51.932572    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-3d8404-dockercfg-kh9h2\""
Nov 02 23:04:51.964350 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:51.964313    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b716a68d-1d7e-48d6-98a5-6fdc5472cac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f\" (UID: \"b716a68d-1d7e-48d6-98a5-6fdc5472cac9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f"
Nov 02 23:04:51.964540 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:51.964466    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7m7p\" (UniqueName: \"kubernetes.io/projected/b716a68d-1d7e-48d6-98a5-6fdc5472cac9-kube-api-access-k7m7p\") pod \"payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f\" (UID: \"b716a68d-1d7e-48d6-98a5-6fdc5472cac9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f"
Nov 02 23:04:51.965074 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:51.965056    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b716a68d-1d7e-48d6-98a5-6fdc5472cac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f\" (UID: \"b716a68d-1d7e-48d6-98a5-6fdc5472cac9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f"
Nov 02 23:04:51.988800 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:51.988762    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh"]
Nov 02 23:04:51.991254 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:51.991235    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh"
Nov 02 23:04:52.018708 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:52.018675    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7m7p\" (UniqueName: \"kubernetes.io/projected/b716a68d-1d7e-48d6-98a5-6fdc5472cac9-kube-api-access-k7m7p\") pod \"payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f\" (UID: \"b716a68d-1d7e-48d6-98a5-6fdc5472cac9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f"
Nov 02 23:04:52.045634 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:52.045602    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh"]
Nov 02 23:04:52.065455 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:52.065415    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8p7p\" (UniqueName: \"kubernetes.io/projected/78e187a5-e905-410b-a2c5-ddced7693786-kube-api-access-f8p7p\") pod \"payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh\" (UID: \"78e187a5-e905-410b-a2c5-ddced7693786\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh"
Nov 02 23:04:52.065685 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:52.065535    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e187a5-e905-410b-a2c5-ddced7693786-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh\" (UID: \"78e187a5-e905-410b-a2c5-ddced7693786\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh"
Nov 02 23:04:52.144257 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:52.144161    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f"
Nov 02 23:04:52.167099 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:52.167063    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8p7p\" (UniqueName: \"kubernetes.io/projected/78e187a5-e905-410b-a2c5-ddced7693786-kube-api-access-f8p7p\") pod \"payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh\" (UID: \"78e187a5-e905-410b-a2c5-ddced7693786\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh"
Nov 02 23:04:52.167248 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:52.167139    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e187a5-e905-410b-a2c5-ddced7693786-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh\" (UID: \"78e187a5-e905-410b-a2c5-ddced7693786\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh"
Nov 02 23:04:52.168014 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:52.167931    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e187a5-e905-410b-a2c5-ddced7693786-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh\" (UID: \"78e187a5-e905-410b-a2c5-ddced7693786\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh"
Nov 02 23:04:52.221294 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:52.221258    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8p7p\" (UniqueName: \"kubernetes.io/projected/78e187a5-e905-410b-a2c5-ddced7693786-kube-api-access-f8p7p\") pod \"payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh\" (UID: \"78e187a5-e905-410b-a2c5-ddced7693786\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh"
Nov 02 23:04:52.303563 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:52.303517    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh"
Nov 02 23:04:52.508376 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:52.508348    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f"]
Nov 02 23:04:52.508893 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:04:52.508866    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb716a68d_1d7e_48d6_98a5_6fdc5472cac9.slice/crio-b394a349200e09e765c04c3939d33dc440e8378c7f05b1030800f4fbec9f2a9a WatchSource:0}: Error finding container b394a349200e09e765c04c3939d33dc440e8378c7f05b1030800f4fbec9f2a9a: Status 404 returned error can't find the container with id b394a349200e09e765c04c3939d33dc440e8378c7f05b1030800f4fbec9f2a9a
Nov 02 23:04:52.668998 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:04:52.668952    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e187a5_e905_410b_a2c5_ddced7693786.slice/crio-3bc2ae83186ed2abe827ad8af4356f01c52a2cd778dcfdef7992b8dbd643b999 WatchSource:0}: Error finding container 3bc2ae83186ed2abe827ad8af4356f01c52a2cd778dcfdef7992b8dbd643b999: Status 404 returned error can't find the container with id 3bc2ae83186ed2abe827ad8af4356f01c52a2cd778dcfdef7992b8dbd643b999
Nov 02 23:04:52.668998 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:52.668971    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh"]
Nov 02 23:04:52.994867 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:52.994825    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f" event={"ID":"b716a68d-1d7e-48d6-98a5-6fdc5472cac9","Type":"ContainerStarted","Data":"c422267b001cf69d1eae9ddb63f2283ee6bbd619e80870d35512f03dc0418783"}
Nov 02 23:04:52.995402 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:52.994888    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f" event={"ID":"b716a68d-1d7e-48d6-98a5-6fdc5472cac9","Type":"ContainerStarted","Data":"b394a349200e09e765c04c3939d33dc440e8378c7f05b1030800f4fbec9f2a9a"}
Nov 02 23:04:52.995402 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:52.994913    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f"
Nov 02 23:04:52.996529 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:52.996490    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh" event={"ID":"78e187a5-e905-410b-a2c5-ddced7693786","Type":"ContainerStarted","Data":"de513cd3a4405ddf97a023f3ed2226e1536e7cef74f34cb24b5527d5b6ec96c6"}
Nov 02 23:04:52.996656 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:52.996548    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh" event={"ID":"78e187a5-e905-410b-a2c5-ddced7693786","Type":"ContainerStarted","Data":"3bc2ae83186ed2abe827ad8af4356f01c52a2cd778dcfdef7992b8dbd643b999"}
Nov 02 23:04:53.105345 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:53.105284    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f" podStartSLOduration=2.105270092 podStartE2EDuration="2.105270092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:53.104090718 +0000 UTC m=+732.672445505" watchObservedRunningTime="2026-11-02 23:04:53.105270092 +0000 UTC m=+732.673624879"
Nov 02 23:04:53.212765 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:53.212707    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh" podStartSLOduration=2.212693219 podStartE2EDuration="2.212693219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:04:53.211387487 +0000 UTC m=+732.779742298" watchObservedRunningTime="2026-11-02 23:04:53.212693219 +0000 UTC m=+732.781048006"
Nov 02 23:04:53.884654 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:53.884619    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:54.002550 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:54.002516    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh"
Nov 02 23:04:57.756000 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:57.755942    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:58.152652 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.152557    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:58.335956 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.335923    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:58.519675 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.519640    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:58.528357 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.528329    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr"]
Nov 02 23:04:58.531077 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.531050    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr"
Nov 02 23:04:58.587171 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.587113    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr"]
Nov 02 23:04:58.631296 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.631267    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-fb6e1d\""
Nov 02 23:04:58.631463 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.631374    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-fb6e1d-dockercfg-nj7l8\""
Nov 02 23:04:58.638799 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.638764    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d7bfa3-6d24-4821-bde2-5272ab59b0c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr\" (UID: \"82d7bfa3-6d24-4821-bde2-5272ab59b0c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr"
Nov 02 23:04:58.638940 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.638806    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pcv4\" (UniqueName: \"kubernetes.io/projected/82d7bfa3-6d24-4821-bde2-5272ab59b0c3-kube-api-access-9pcv4\") pod \"payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr\" (UID: \"82d7bfa3-6d24-4821-bde2-5272ab59b0c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr"
Nov 02 23:04:58.685628 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.685595    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b"]
Nov 02 23:04:58.688167 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.688150    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b"
Nov 02 23:04:58.740741 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.740093    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f480410-faab-4712-b042-4e02d5759d11-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b\" (UID: \"6f480410-faab-4712-b042-4e02d5759d11\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b"
Nov 02 23:04:58.740741 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.740168    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k527k\" (UniqueName: \"kubernetes.io/projected/6f480410-faab-4712-b042-4e02d5759d11-kube-api-access-k527k\") pod \"payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b\" (UID: \"6f480410-faab-4712-b042-4e02d5759d11\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b"
Nov 02 23:04:58.740741 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.740320    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d7bfa3-6d24-4821-bde2-5272ab59b0c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr\" (UID: \"82d7bfa3-6d24-4821-bde2-5272ab59b0c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr"
Nov 02 23:04:58.740741 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.740356    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pcv4\" (UniqueName: \"kubernetes.io/projected/82d7bfa3-6d24-4821-bde2-5272ab59b0c3-kube-api-access-9pcv4\") pod \"payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr\" (UID: \"82d7bfa3-6d24-4821-bde2-5272ab59b0c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr"
Nov 02 23:04:58.741499 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.741474    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d7bfa3-6d24-4821-bde2-5272ab59b0c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr\" (UID: \"82d7bfa3-6d24-4821-bde2-5272ab59b0c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr"
Nov 02 23:04:58.744696 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.744661    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b"]
Nov 02 23:04:58.797347 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.797272    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pcv4\" (UniqueName: \"kubernetes.io/projected/82d7bfa3-6d24-4821-bde2-5272ab59b0c3-kube-api-access-9pcv4\") pod \"payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr\" (UID: \"82d7bfa3-6d24-4821-bde2-5272ab59b0c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr"
Nov 02 23:04:58.822970 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.822935    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:58.841257 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.841223    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f480410-faab-4712-b042-4e02d5759d11-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b\" (UID: \"6f480410-faab-4712-b042-4e02d5759d11\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b"
Nov 02 23:04:58.841431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.841274    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k527k\" (UniqueName: \"kubernetes.io/projected/6f480410-faab-4712-b042-4e02d5759d11-kube-api-access-k527k\") pod \"payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b\" (UID: \"6f480410-faab-4712-b042-4e02d5759d11\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b"
Nov 02 23:04:58.842115 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.842095    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f480410-faab-4712-b042-4e02d5759d11-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b\" (UID: \"6f480410-faab-4712-b042-4e02d5759d11\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b"
Nov 02 23:04:58.845151 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.845121    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr"
Nov 02 23:04:58.898141 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.898080    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k527k\" (UniqueName: \"kubernetes.io/projected/6f480410-faab-4712-b042-4e02d5759d11-kube-api-access-k527k\") pod \"payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b\" (UID: \"6f480410-faab-4712-b042-4e02d5759d11\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b"
Nov 02 23:04:58.988014 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:58.987968    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:59.001380 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:59.001352    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b"
Nov 02 23:04:59.157315 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:59.157268    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:04:59.222952 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:59.220373    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr"]
Nov 02 23:04:59.224078 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:04:59.224045    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d7bfa3_6d24_4821_bde2_5272ab59b0c3.slice/crio-649ce0f273e4f3ad2b0fda57f8ada58cd393ee6d6f4f858864583d4662e7d888 WatchSource:0}: Error finding container 649ce0f273e4f3ad2b0fda57f8ada58cd393ee6d6f4f858864583d4662e7d888: Status 404 returned error can't find the container with id 649ce0f273e4f3ad2b0fda57f8ada58cd393ee6d6f4f858864583d4662e7d888
Nov 02 23:04:59.372244 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:59.372211    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b"]
Nov 02 23:04:59.373697 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:04:59.373667    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f480410_faab_4712_b042_4e02d5759d11.slice/crio-4deda747dcc4e119e42cbe78821f55c99f31d845c19dad31506c075622101282 WatchSource:0}: Error finding container 4deda747dcc4e119e42cbe78821f55c99f31d845c19dad31506c075622101282: Status 404 returned error can't find the container with id 4deda747dcc4e119e42cbe78821f55c99f31d845c19dad31506c075622101282
Nov 02 23:04:59.977946 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:59.977915    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj"
Nov 02 23:04:59.978446 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:04:59.978013    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7"
Nov 02 23:05:00.041611 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:00.041198    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b" event={"ID":"6f480410-faab-4712-b042-4e02d5759d11","Type":"ContainerStarted","Data":"65abc4b4e8e5fb7a304380ca64c9681f8d95670734fe8e711b4bc062ae61a9fe"}
Nov 02 23:05:00.041611 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:00.041261    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b" event={"ID":"6f480410-faab-4712-b042-4e02d5759d11","Type":"ContainerStarted","Data":"4deda747dcc4e119e42cbe78821f55c99f31d845c19dad31506c075622101282"}
Nov 02 23:05:00.041611 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:00.041306    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b"
Nov 02 23:05:00.043282 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:00.043256    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr" event={"ID":"82d7bfa3-6d24-4821-bde2-5272ab59b0c3","Type":"ContainerStarted","Data":"e80be45d5d272679e8ac8c1712067ba42a7c3ab8a6ed4ce73ec212d018b37751"}
Nov 02 23:05:00.043419 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:00.043301    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr"
Nov 02 23:05:00.043419 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:00.043318    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr" event={"ID":"82d7bfa3-6d24-4821-bde2-5272ab59b0c3","Type":"ContainerStarted","Data":"649ce0f273e4f3ad2b0fda57f8ada58cd393ee6d6f4f858864583d4662e7d888"}
Nov 02 23:05:00.247941 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:00.247909    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:05:00.261656 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:00.261623    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:05:00.303277 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:00.303072    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b" podStartSLOduration=2.303054604 podStartE2EDuration="2.303054604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:00.302256555 +0000 UTC m=+739.870611342" watchObservedRunningTime="2026-11-02 23:05:00.303054604 +0000 UTC m=+739.871409393"
Nov 02 23:05:00.412893 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:00.412825    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr" podStartSLOduration=2.412804102 podStartE2EDuration="2.412804102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:00.410625663 +0000 UTC m=+739.978980475" watchObservedRunningTime="2026-11-02 23:05:00.412804102 +0000 UTC m=+739.981158888"
Nov 02 23:05:01.059771 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:01.059740    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:05:03.293617 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:03.293581    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"]
Nov 02 23:05:03.296336 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:03.296318    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"
Nov 02 23:05:03.385736 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:03.385693    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g9k2\" (UniqueName: \"kubernetes.io/projected/a2210e3e-ee40-4277-8865-93acdf8e81a4-kube-api-access-4g9k2\") pod \"maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn\" (UID: \"a2210e3e-ee40-4277-8865-93acdf8e81a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"
Nov 02 23:05:03.385917 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:03.385747    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2210e3e-ee40-4277-8865-93acdf8e81a4-metrics-certs\") pod \"maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn\" (UID: \"a2210e3e-ee40-4277-8865-93acdf8e81a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"
Nov 02 23:05:03.385917 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:03.385831    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2210e3e-ee40-4277-8865-93acdf8e81a4-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn\" (UID: \"a2210e3e-ee40-4277-8865-93acdf8e81a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"
Nov 02 23:05:03.385917 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:03.385897    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2210e3e-ee40-4277-8865-93acdf8e81a4-maas-api-tls\") pod \"maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn\" (UID: \"a2210e3e-ee40-4277-8865-93acdf8e81a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"
Nov 02 23:05:03.395136 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:03.395108    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-3d8404-serving-cert\""
Nov 02 23:05:03.397747 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:03.397722    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"]
Nov 02 23:05:03.487552 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:03.487511    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g9k2\" (UniqueName: \"kubernetes.io/projected/a2210e3e-ee40-4277-8865-93acdf8e81a4-kube-api-access-4g9k2\") pod \"maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn\" (UID: \"a2210e3e-ee40-4277-8865-93acdf8e81a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"
Nov 02 23:05:03.487552 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:03.487566    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2210e3e-ee40-4277-8865-93acdf8e81a4-metrics-certs\") pod \"maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn\" (UID: \"a2210e3e-ee40-4277-8865-93acdf8e81a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"
Nov 02 23:05:03.487788 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:03.487681    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2210e3e-ee40-4277-8865-93acdf8e81a4-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn\" (UID: \"a2210e3e-ee40-4277-8865-93acdf8e81a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"
Nov 02 23:05:03.487788 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:03.487763    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2210e3e-ee40-4277-8865-93acdf8e81a4-maas-api-tls\") pod \"maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn\" (UID: \"a2210e3e-ee40-4277-8865-93acdf8e81a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"
Nov 02 23:05:03.487894 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:05:03.487880    4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-serving-cert: secret "maas-api-e2e-worker-w1-3d8404-serving-cert" not found
Nov 02 23:05:03.487947 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:05:03.487938    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2210e3e-ee40-4277-8865-93acdf8e81a4-maas-api-tls podName:a2210e3e-ee40-4277-8865-93acdf8e81a4 nodeName:}" failed. No retries permitted until 2026-11-02 23:05:03.987922514 +0000 UTC m=+743.556277279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2210e3e-ee40-4277-8865-93acdf8e81a4-maas-api-tls") pod "maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn" (UID: "a2210e3e-ee40-4277-8865-93acdf8e81a4") : secret "maas-api-e2e-worker-w1-3d8404-serving-cert" not found
Nov 02 23:05:03.488513 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:03.488482    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2210e3e-ee40-4277-8865-93acdf8e81a4-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn\" (UID: \"a2210e3e-ee40-4277-8865-93acdf8e81a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"
Nov 02 23:05:03.490300 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:03.490272    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2210e3e-ee40-4277-8865-93acdf8e81a4-metrics-certs\") pod \"maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn\" (UID: \"a2210e3e-ee40-4277-8865-93acdf8e81a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"
Nov 02 23:05:03.515540 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:03.515502    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:03.544808 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:03.544727    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g9k2\" (UniqueName: \"kubernetes.io/projected/a2210e3e-ee40-4277-8865-93acdf8e81a4-kube-api-access-4g9k2\") pod \"maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn\" (UID: \"a2210e3e-ee40-4277-8865-93acdf8e81a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"
Nov 02 23:05:03.691255 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:03.691217    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:03.848114 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:03.848024    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:03.993286 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:03.993254    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2210e3e-ee40-4277-8865-93acdf8e81a4-maas-api-tls\") pod \"maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn\" (UID: \"a2210e3e-ee40-4277-8865-93acdf8e81a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"
Nov 02 23:05:03.995736 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:03.995713    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2210e3e-ee40-4277-8865-93acdf8e81a4-maas-api-tls\") pod \"maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn\" (UID: \"a2210e3e-ee40-4277-8865-93acdf8e81a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"
Nov 02 23:05:04.006233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:04.006205    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f"
Nov 02 23:05:04.208477 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:04.208388    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"
Nov 02 23:05:04.620818 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:04.620786    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"]
Nov 02 23:05:04.621286 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:05:04.621004    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2210e3e_ee40_4277_8865_93acdf8e81a4.slice/crio-a5b6291af0ce59ba45615b60cd29e7b65faec9e237ffbd27ffd0b5f5d5334d57 WatchSource:0}: Error finding container a5b6291af0ce59ba45615b60cd29e7b65faec9e237ffbd27ffd0b5f5d5334d57: Status 404 returned error can't find the container with id a5b6291af0ce59ba45615b60cd29e7b65faec9e237ffbd27ffd0b5f5d5334d57
Nov 02 23:05:04.622837 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:04.622816    4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 23:05:05.010515 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:05.010482    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh"
Nov 02 23:05:05.071032 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:05.070991    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn" event={"ID":"a2210e3e-ee40-4277-8865-93acdf8e81a4","Type":"ContainerStarted","Data":"a5b6291af0ce59ba45615b60cd29e7b65faec9e237ffbd27ffd0b5f5d5334d57"}
Nov 02 23:05:06.085962 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:06.085921    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn" event={"ID":"a2210e3e-ee40-4277-8865-93acdf8e81a4","Type":"ContainerStarted","Data":"1e7984386ece16c38a92cdd927b5cbb0968c427e76e94ddd9626c4f3b1f39654"}
Nov 02 23:05:06.086409 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:06.086041    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"
Nov 02 23:05:06.195193 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:06.195131    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn" podStartSLOduration=2.276793189 podStartE2EDuration="3.195119318s" totalImagesPullingTime="918.326129ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:06.194776746 +0000 UTC m=+745.763131533" watchObservedRunningTime="2026-11-02 23:05:06.195119318 +0000 UTC m=+745.763474105"
Nov 02 23:05:08.161314 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:08.161278    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:11.051022 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:11.050993    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b"
Nov 02 23:05:11.051626 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:11.051610    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr"
Nov 02 23:05:11.751326 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:05:11.751282    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:05:12.106837 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:12.106742    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"
Nov 02 23:05:12.702735 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:12.702687    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:12.874006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:12.870551    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:13.027041 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:13.026971    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:13.735016 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:13.734969    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:17.439896 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:17.439859    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:18.155018 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:18.154965    4294 generic.go:356] "Generic (PLEG): container finished" podID="da283988-fca2-4bde-8256-4589d9536e4b" containerID="d1735b7f2e08e253b049e20bb5e5d05d34d35673bc7c0a8d396e6fc6a9af6c8c" exitCode=0
Nov 02 23:05:18.155281 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:18.155012    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29" event={"ID":"da283988-fca2-4bde-8256-4589d9536e4b","Type":"ContainerDied","Data":"d1735b7f2e08e253b049e20bb5e5d05d34d35673bc7c0a8d396e6fc6a9af6c8c"}
Nov 02 23:05:18.155475 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:18.155458    4294 scope.go:122] "RemoveContainer" containerID="d1735b7f2e08e253b049e20bb5e5d05d34d35673bc7c0a8d396e6fc6a9af6c8c"
Nov 02 23:05:18.156886 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:18.156859    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:05:18.156940 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:18.156930    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:05:18.156999 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:18.156960    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:05:18.204498 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:18.204464    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:19.186720 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:19.186685    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29" event={"ID":"da283988-fca2-4bde-8256-4589d9536e4b","Type":"ContainerStarted","Data":"54cb2277ba4e27b198d663aba7731387d3efbb212569dea636b45c3eda572a06"}
Nov 02 23:05:19.188843 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:19.188805    4294 generic.go:356] "Generic (PLEG): container finished" podID="4571d9d7-7554-444c-a4d7-ba0daf1006fc" containerID="e1ac82ff6b0c8f8219d1c78859909cddb14aefb2fd130246557b889e7dd143a5" exitCode=0
Nov 02 23:05:19.189017 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:19.188861    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr" event={"ID":"4571d9d7-7554-444c-a4d7-ba0daf1006fc","Type":"ContainerDied","Data":"e1ac82ff6b0c8f8219d1c78859909cddb14aefb2fd130246557b889e7dd143a5"}
Nov 02 23:05:19.189318 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:19.189302    4294 scope.go:122] "RemoveContainer" containerID="e1ac82ff6b0c8f8219d1c78859909cddb14aefb2fd130246557b889e7dd143a5"
Nov 02 23:05:19.190849 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:19.190824    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:05:19.190916 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:19.190903    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:05:19.190951 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:19.190942    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:05:19.260710 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:19.260447    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:05:19.260710 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:19.260593    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 02 23:05:19.260710 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:19.260643    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29" podUID="da283988-fca2-4bde-8256-4589d9536e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 02 23:05:20.224006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:20.223040    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr" event={"ID":"4571d9d7-7554-444c-a4d7-ba0daf1006fc","Type":"ContainerStarted","Data":"6efdc662aac95b84e6ff6b739b6eba3e81dcc129866440e5b65357be01202897"}
Nov 02 23:05:20.262005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:20.257529    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 02 23:05:20.262005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:20.257608    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29" podUID="da283988-fca2-4bde-8256-4589d9536e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 02 23:05:20.991450 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:20.991407    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:05:20.992607 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:20.991743    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 23:05:20.992816 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:20.992791    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr" podUID="4571d9d7-7554-444c-a4d7-ba0daf1006fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 23:05:21.261535 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:21.261435    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 02 23:05:21.271006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:21.266068    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29" podUID="da283988-fca2-4bde-8256-4589d9536e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 02 23:05:21.990676 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:21.990635    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 23:05:21.991370 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:21.990938    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr" podUID="4571d9d7-7554-444c-a4d7-ba0daf1006fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 23:05:22.044463 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:22.044431    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:22.258548 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:22.258445    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 02 23:05:22.258817 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:22.258783    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29" podUID="da283988-fca2-4bde-8256-4589d9536e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 02 23:05:22.990478 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:22.990431    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 23:05:22.991126 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:22.991088    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr" podUID="4571d9d7-7554-444c-a4d7-ba0daf1006fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 23:05:23.262014 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:23.257556    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 02 23:05:23.262014 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:23.257635    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29" podUID="da283988-fca2-4bde-8256-4589d9536e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 02 23:05:23.994004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:23.993422    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 23:05:23.994004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:23.993487    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr" podUID="4571d9d7-7554-444c-a4d7-ba0daf1006fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 23:05:24.257531 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:24.257427    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 02 23:05:24.257796 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:24.257762    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29" podUID="da283988-fca2-4bde-8256-4589d9536e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 02 23:05:24.993546 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:24.993507    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 02 23:05:24.993842 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:24.993798    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr" podUID="4571d9d7-7554-444c-a4d7-ba0daf1006fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 02 23:05:25.112148 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:25.112112    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:25.271059 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:25.269067    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:05:25.271059 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:25.270468    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:05:25.278334 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:25.278287    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 02 23:05:25.955840 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:25.955805    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:25.995472 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:25.995441    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:05:25.995761 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:25.995741    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:05:25.996581 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:25.996559    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:05:26.102863 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:26.102805    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:26.546265 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:26.546228    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:27.084481 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:27.084436    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:29.306727 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:29.306692    4294 generic.go:356] "Generic (PLEG): container finished" podID="9d525079-fdb8-4c0e-bb38-2567fcf4a49a" containerID="58c2c1ee3f93ef3829295a5eee14912143845af9968d63c703eea9ac2c5a9311" exitCode=0
Nov 02 23:05:29.307188 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:29.306769    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp" event={"ID":"9d525079-fdb8-4c0e-bb38-2567fcf4a49a","Type":"ContainerDied","Data":"58c2c1ee3f93ef3829295a5eee14912143845af9968d63c703eea9ac2c5a9311"}
Nov 02 23:05:29.307248 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:29.307228    4294 scope.go:122] "RemoveContainer" containerID="58c2c1ee3f93ef3829295a5eee14912143845af9968d63c703eea9ac2c5a9311"
Nov 02 23:05:29.308481 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:29.308454    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:05:29.308539 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:29.308523    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:05:29.308752 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:29.308554    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:05:29.937637 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:29.937601    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:05:30.314232 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:30.314191    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp" event={"ID":"9d525079-fdb8-4c0e-bb38-2567fcf4a49a","Type":"ContainerStarted","Data":"fbd56b7432201d2d341e246841a0cccaa25826338200dcdaabf3688945d3b21b"}
Nov 02 23:05:31.052162 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:31.052132    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:31.134532 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:31.134494    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:05:31.136055 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:31.136022    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 23:05:31.136222 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:31.136079    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp" podUID="9d525079-fdb8-4c0e-bb38-2567fcf4a49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 23:05:32.134168 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:32.134131    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 23:05:32.134596 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:32.134188    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp" podUID="9d525079-fdb8-4c0e-bb38-2567fcf4a49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 23:05:33.134369 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:33.134324    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 23:05:33.134791 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:33.134399    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp" podUID="9d525079-fdb8-4c0e-bb38-2567fcf4a49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 23:05:33.346880 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:33.346843    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:33.921748 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:33.921707    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:34.091841 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:34.091799    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:34.134506 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:34.134467    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 23:05:34.134905 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:34.134538    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp" podUID="9d525079-fdb8-4c0e-bb38-2567fcf4a49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 23:05:35.134399 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:35.134361    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 02 23:05:35.134607 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:35.134439    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp" podUID="9d525079-fdb8-4c0e-bb38-2567fcf4a49a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 02 23:05:36.070782 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:36.070743    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:36.138797 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:36.138770    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:05:36.139237 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:36.139100    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:05:36.139954 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:36.139933    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:05:39.456445 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:39.456389    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:40.119159 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:40.119119    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:40.297117 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:40.297082    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:41.065643 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:41.065607    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:41.749869 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:05:41.749823    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:05:45.667447 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:45.667402    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:47.416398 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.416363    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"]
Nov 02 23:05:47.419665 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.419646    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.446374 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.446341    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:47.465832 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.465794    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-istio-envoy\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.465832 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.465834    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-istio-data\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.466079 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.466014    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-workload-certs\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.466079 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.466051    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96fc6247-0b37-40df-b795-2c799c981876-istiod-ca-cert\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.466150 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.466081    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96fc6247-0b37-40df-b795-2c799c981876-istio-token\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.466183 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.466147    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-workload-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.466183 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.466170    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9qxz\" (UniqueName: \"kubernetes.io/projected/96fc6247-0b37-40df-b795-2c799c981876-kube-api-access-f9qxz\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.466256 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.466201    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96fc6247-0b37-40df-b795-2c799c981876-istio-podinfo\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.466256 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.466242    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-credential-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.519458 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.519419    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-14b321-openshift-default-dockercfg-ssm6b\""
Nov 02 23:05:47.522124 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.522101    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"]
Nov 02 23:05:47.567495 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.567448    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-workload-certs\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.567708 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.567510    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96fc6247-0b37-40df-b795-2c799c981876-istiod-ca-cert\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.567708 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.567532    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96fc6247-0b37-40df-b795-2c799c981876-istio-token\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.567708 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.567565    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-workload-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.567890 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.567779    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9qxz\" (UniqueName: \"kubernetes.io/projected/96fc6247-0b37-40df-b795-2c799c981876-kube-api-access-f9qxz\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.567890 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.567832    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96fc6247-0b37-40df-b795-2c799c981876-istio-podinfo\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.567890 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.567863    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-credential-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.568094 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.567919    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-istio-envoy\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.568094 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.567930    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-workload-certs\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.568094 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.567938    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-workload-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.568094 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.567940    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-istio-data\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.568298 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.568229    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-istio-data\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.568353 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.568300    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-credential-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.568415 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.568386    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96fc6247-0b37-40df-b795-2c799c981876-istiod-ca-cert\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.570241 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.570203    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-istio-envoy\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.570498 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.570478    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96fc6247-0b37-40df-b795-2c799c981876-istio-podinfo\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.612733 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.612704    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:47.625326 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.625296    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96fc6247-0b37-40df-b795-2c799c981876-istio-token\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.625494 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.625376    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9qxz\" (UniqueName: \"kubernetes.io/projected/96fc6247-0b37-40df-b795-2c799c981876-kube-api-access-f9qxz\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:47.731845 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:47.731742    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:48.176395 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:05:48.176363    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96fc6247_0b37_40df_b795_2c799c981876.slice/crio-51b08f87026af2f02059867c823694c1aff9b977eb1c648dab2c1abb4cc3bf80 WatchSource:0}: Error finding container 51b08f87026af2f02059867c823694c1aff9b977eb1c648dab2c1abb4cc3bf80: Status 404 returned error can't find the container with id 51b08f87026af2f02059867c823694c1aff9b977eb1c648dab2c1abb4cc3bf80
Nov 02 23:05:48.176721 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:48.176697    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"]
Nov 02 23:05:48.179262 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:48.179231    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:05:48.179380 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:48.179297    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:05:48.179380 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:48.179335    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:05:48.212934 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:48.212901    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:48.378341 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:48.378308    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:48.420027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:48.419993    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl" event={"ID":"96fc6247-0b37-40df-b795-2c799c981876","Type":"ContainerStarted","Data":"c0d05befac874f4db919144cda4523025522f5347d227f20168f0a4ff352e1dd"}
Nov 02 23:05:48.420527 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:48.420050    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl" event={"ID":"96fc6247-0b37-40df-b795-2c799c981876","Type":"ContainerStarted","Data":"51b08f87026af2f02059867c823694c1aff9b977eb1c648dab2c1abb4cc3bf80"}
Nov 02 23:05:48.732419 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:48.732372    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:49.733824 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:49.733790    4294 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 23:05:49.734256 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:49.733846    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl" podUID="96fc6247-0b37-40df-b795-2c799c981876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 23:05:50.579902 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:50.579864    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:50.732831 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:50.732792    4294 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 23:05:50.733037 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:50.732848    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl" podUID="96fc6247-0b37-40df-b795-2c799c981876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 23:05:51.732444 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:51.732398    4294 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 23:05:51.732930 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:51.732467    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl" podUID="96fc6247-0b37-40df-b795-2c799c981876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 23:05:52.732329 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:52.732294    4294 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 23:05:52.732520 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:52.732349    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl" podUID="96fc6247-0b37-40df-b795-2c799c981876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 23:05:53.732464 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:53.732428    4294 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 23:05:53.732665 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:53.732490    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl" podUID="96fc6247-0b37-40df-b795-2c799c981876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 23:05:54.732735 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:54.732691    4294 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Nov 02 23:05:54.733158 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:54.732769    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl" podUID="96fc6247-0b37-40df-b795-2c799c981876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Nov 02 23:05:55.157824 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:55.157783    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:05:55.737273 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:55.737244    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:55.737734 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:55.737719    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:55.738441 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:55.738409    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:05:55.897789 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:55.897731    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl" podStartSLOduration=8.897716227 podStartE2EDuration="8.897716227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:05:48.537797352 +0000 UTC m=+788.106152141" watchObservedRunningTime="2026-11-02 23:05:55.897716227 +0000 UTC m=+795.466071068"
Nov 02 23:05:59.775627 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:05:59.775574    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:06:04.347871 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:04.347840    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:06:05.554957 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.554920    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"]
Nov 02 23:06:05.557767 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.557750    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:05.656152 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.656109    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3d8404\"/\"kube-root-ca.crt\""
Nov 02 23:06:05.656152 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.656146    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3d8404\"/\"default-dockercfg-8s6dn\""
Nov 02 23:06:05.656456 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.656169    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3d8404\"/\"openshift-service-ca.crt\""
Nov 02 23:06:05.656456 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.656212    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3d8404\"/\"simulator-w1-3d8404-kserve-self-signed-certs\""
Nov 02 23:06:05.658592 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.658566    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"]
Nov 02 23:06:05.677668 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.677617    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:06:05.679866 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.679841    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6kq6\" (UniqueName: \"kubernetes.io/projected/2f110c17-1134-4f5c-b9a6-51abb2ac481f-kube-api-access-m6kq6\") pod \"simulator-w1-3d8404-kserve-6d6477c969-hz6jf\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") " pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:05.680021 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.679879    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-dshm\") pod \"simulator-w1-3d8404-kserve-6d6477c969-hz6jf\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") " pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:05.680021 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.679901    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-home\") pod \"simulator-w1-3d8404-kserve-6d6477c969-hz6jf\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") " pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:05.680112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.680072    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-model-cache\") pod \"simulator-w1-3d8404-kserve-6d6477c969-hz6jf\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") " pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:05.680163 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.680136    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f110c17-1134-4f5c-b9a6-51abb2ac481f-tls-certs\") pod \"simulator-w1-3d8404-kserve-6d6477c969-hz6jf\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") " pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:05.680234 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.680222    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-tmp-dir\") pod \"simulator-w1-3d8404-kserve-6d6477c969-hz6jf\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") " pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:05.781346 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.781312    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f110c17-1134-4f5c-b9a6-51abb2ac481f-tls-certs\") pod \"simulator-w1-3d8404-kserve-6d6477c969-hz6jf\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") " pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:05.781346 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.781356    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-tmp-dir\") pod \"simulator-w1-3d8404-kserve-6d6477c969-hz6jf\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") " pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:05.781586 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.781415    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6kq6\" (UniqueName: \"kubernetes.io/projected/2f110c17-1134-4f5c-b9a6-51abb2ac481f-kube-api-access-m6kq6\") pod \"simulator-w1-3d8404-kserve-6d6477c969-hz6jf\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") " pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:05.781586 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.781452    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-dshm\") pod \"simulator-w1-3d8404-kserve-6d6477c969-hz6jf\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") " pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:05.781586 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.781481    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-home\") pod \"simulator-w1-3d8404-kserve-6d6477c969-hz6jf\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") " pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:05.781586 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.781570    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-model-cache\") pod \"simulator-w1-3d8404-kserve-6d6477c969-hz6jf\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") " pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:05.781919 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.781890    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-tmp-dir\") pod \"simulator-w1-3d8404-kserve-6d6477c969-hz6jf\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") " pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:05.782069 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.781902    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-home\") pod \"simulator-w1-3d8404-kserve-6d6477c969-hz6jf\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") " pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:05.782131 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.782082    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-model-cache\") pod \"simulator-w1-3d8404-kserve-6d6477c969-hz6jf\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") " pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:05.783831 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.783809    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-dshm\") pod \"simulator-w1-3d8404-kserve-6d6477c969-hz6jf\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") " pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:05.784146 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.784129    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f110c17-1134-4f5c-b9a6-51abb2ac481f-tls-certs\") pod \"simulator-w1-3d8404-kserve-6d6477c969-hz6jf\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") " pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:05.836015 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.835919    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6kq6\" (UniqueName: \"kubernetes.io/projected/2f110c17-1134-4f5c-b9a6-51abb2ac481f-kube-api-access-m6kq6\") pod \"simulator-w1-3d8404-kserve-6d6477c969-hz6jf\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") " pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:05.869346 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:05.869309    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:06.239213 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:06.239178    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"]
Nov 02 23:06:06.240173 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:06:06.240149    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f110c17_1134_4f5c_b9a6_51abb2ac481f.slice/crio-664f183492a9e7859f7ca1873a2117c0668959b99eec0d4b6686ba4d61c5bb1a WatchSource:0}: Error finding container 664f183492a9e7859f7ca1873a2117c0668959b99eec0d4b6686ba4d61c5bb1a: Status 404 returned error can't find the container with id 664f183492a9e7859f7ca1873a2117c0668959b99eec0d4b6686ba4d61c5bb1a
Nov 02 23:06:06.529276 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:06.529238    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf" event={"ID":"2f110c17-1134-4f5c-b9a6-51abb2ac481f","Type":"ContainerStarted","Data":"e612f806cc9137142b79e598c2ad6c2e82273cae6b39de891d25a1047e05c7c4"}
Nov 02 23:06:06.529464 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:06.529291    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf" event={"ID":"2f110c17-1134-4f5c-b9a6-51abb2ac481f","Type":"ContainerStarted","Data":"664f183492a9e7859f7ca1873a2117c0668959b99eec0d4b6686ba4d61c5bb1a"}
Nov 02 23:06:06.639379 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:06.639325    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf" podStartSLOduration=1.639310187 podStartE2EDuration="1.639310187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:06:06.638360378 +0000 UTC m=+806.206715168" watchObservedRunningTime="2026-11-02 23:06:06.639310187 +0000 UTC m=+806.207664974"
Nov 02 23:06:06.870335 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:06.870238    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:08.947546 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:08.947510    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:06:11.750727 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:06:11.750685    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:06:13.354745 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:13.354707    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:06:13.521166 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:13.521112    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:06:15.870133 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:15.870097    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:15.883226 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:15.883186    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:16.590259 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:16.590231    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:06:18.145942 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:18.145908    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:06:20.500854 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.500820    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"]
Nov 02 23:06:20.503794 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.503777    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:20.552720 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.552672    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-dshm\") pod \"simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") " pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:20.552909 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.552755    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e3d0d6-1a8e-4536-bc46-b5785124405a-tls-certs\") pod \"simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") " pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:20.552909 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.552785    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-model-cache\") pod \"simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") " pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:20.552909 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.552811    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-tmp-dir\") pod \"simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") " pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:20.552909 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.552844    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-home\") pod \"simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") " pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:20.553176 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.552911    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fphlq\" (UniqueName: \"kubernetes.io/projected/43e3d0d6-1a8e-4536-bc46-b5785124405a-kube-api-access-fphlq\") pod \"simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") " pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:20.587999 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.587769    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fb6e1d\"/\"kube-root-ca.crt\""
Nov 02 23:06:20.601593 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.601560    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fb6e1d\"/\"openshift-service-ca.crt\""
Nov 02 23:06:20.601593 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.601560    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fb6e1d\"/\"default-dockercfg-66dv6\""
Nov 02 23:06:20.604047 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.604022    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fb6e1d\"/\"simulator-w4-fb6e1d-kserve-self-signed-certs\""
Nov 02 23:06:20.606605 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.606579    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"]
Nov 02 23:06:20.621989 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.621954    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:06:20.653922 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.653881    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-dshm\") pod \"simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") " pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:20.654119 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.653941    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e3d0d6-1a8e-4536-bc46-b5785124405a-tls-certs\") pod \"simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") " pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:20.654119 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.653960    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-model-cache\") pod \"simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") " pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:20.654119 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.654008    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-tmp-dir\") pod \"simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") " pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:20.654119 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.654038    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-home\") pod \"simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") " pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:20.654119 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.654091    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fphlq\" (UniqueName: \"kubernetes.io/projected/43e3d0d6-1a8e-4536-bc46-b5785124405a-kube-api-access-fphlq\") pod \"simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") " pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:20.654445 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.654422    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-model-cache\") pod \"simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") " pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:20.654575 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.654484    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-home\") pod \"simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") " pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:20.654575 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.654523    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-tmp-dir\") pod \"simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") " pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:20.656410 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.656381    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-dshm\") pod \"simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") " pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:20.656585 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.656569    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e3d0d6-1a8e-4536-bc46-b5785124405a-tls-certs\") pod \"simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") " pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:20.708760 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.708722    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fphlq\" (UniqueName: \"kubernetes.io/projected/43e3d0d6-1a8e-4536-bc46-b5785124405a-kube-api-access-fphlq\") pod \"simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") " pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:20.814894 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:20.814811    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:21.191787 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:21.191754    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"]
Nov 02 23:06:21.192689 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:06:21.192661    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43e3d0d6_1a8e_4536_bc46_b5785124405a.slice/crio-f505f084b1b63470789380d2cb9ea0fded396d9df49c2b2bde1a043b2a2af07d WatchSource:0}: Error finding container f505f084b1b63470789380d2cb9ea0fded396d9df49c2b2bde1a043b2a2af07d: Status 404 returned error can't find the container with id f505f084b1b63470789380d2cb9ea0fded396d9df49c2b2bde1a043b2a2af07d
Nov 02 23:06:21.611901 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:21.611864    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9" event={"ID":"43e3d0d6-1a8e-4536-bc46-b5785124405a","Type":"ContainerStarted","Data":"43911fcc2bc6f85a5cab5a2d21c83dc1b186416ec975aaa39a5a3e07c584cfab"}
Nov 02 23:06:21.611901 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:21.611909    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9" event={"ID":"43e3d0d6-1a8e-4536-bc46-b5785124405a","Type":"ContainerStarted","Data":"f505f084b1b63470789380d2cb9ea0fded396d9df49c2b2bde1a043b2a2af07d"}
Nov 02 23:06:21.740125 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:21.740057    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9" podStartSLOduration=1.740044175 podStartE2EDuration="1.740044175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:06:21.737178362 +0000 UTC m=+821.305533151" watchObservedRunningTime="2026-11-02 23:06:21.740044175 +0000 UTC m=+821.308398962"
Nov 02 23:06:21.815396 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:21.815357    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:22.751172 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:22.751115    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:06:25.422377 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:25.422339    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:06:27.251816 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:27.251782    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:06:30.815043 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:30.815009    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:30.829280 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:30.829250    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:31.685099 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:31.685065    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:06:31.866028 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:31.865994    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:06:36.447902 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:36.447866    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:06:41.068504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:41.068466    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:06:41.751860 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:06:41.751822    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:06:43.065745 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.065712    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"]
Nov 02 23:06:43.068593 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.068577    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:43.101293 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.101258    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-home\") pod \"premium-w2-ba1918-kserve-85864b4646-nxprb\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") " pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:43.101464 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.101305    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpfj6\" (UniqueName: \"kubernetes.io/projected/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-kube-api-access-mpfj6\") pod \"premium-w2-ba1918-kserve-85864b4646-nxprb\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") " pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:43.101528 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.101481    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-model-cache\") pod \"premium-w2-ba1918-kserve-85864b4646-nxprb\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") " pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:43.101570 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.101534    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-tls-certs\") pod \"premium-w2-ba1918-kserve-85864b4646-nxprb\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") " pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:43.101636 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.101617    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-tmp-dir\") pod \"premium-w2-ba1918-kserve-85864b4646-nxprb\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") " pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:43.101686 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.101671    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-dshm\") pod \"premium-w2-ba1918-kserve-85864b4646-nxprb\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") " pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:43.166897 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.166859    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ba1918\"/\"kube-root-ca.crt\""
Nov 02 23:06:43.167123 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.167104    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ba1918\"/\"openshift-service-ca.crt\""
Nov 02 23:06:43.167178 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.167103    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ba1918\"/\"default-dockercfg-gpvcd\""
Nov 02 23:06:43.167490 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.167477    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ba1918\"/\"premium-w2-ba1918-kserve-self-signed-certs\""
Nov 02 23:06:43.169808 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.169787    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"]
Nov 02 23:06:43.185642 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.185615    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:06:43.203123 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.203084    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-model-cache\") pod \"premium-w2-ba1918-kserve-85864b4646-nxprb\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") " pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:43.203123 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.203126    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-tls-certs\") pod \"premium-w2-ba1918-kserve-85864b4646-nxprb\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") " pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:43.203370 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.203153    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-tmp-dir\") pod \"premium-w2-ba1918-kserve-85864b4646-nxprb\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") " pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:43.203370 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.203190    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-dshm\") pod \"premium-w2-ba1918-kserve-85864b4646-nxprb\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") " pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:43.203370 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.203300    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-home\") pod \"premium-w2-ba1918-kserve-85864b4646-nxprb\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") " pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:43.203370 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.203325    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpfj6\" (UniqueName: \"kubernetes.io/projected/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-kube-api-access-mpfj6\") pod \"premium-w2-ba1918-kserve-85864b4646-nxprb\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") " pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:43.203668 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.203645    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-model-cache\") pod \"premium-w2-ba1918-kserve-85864b4646-nxprb\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") " pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:43.203763 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.203696    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-tmp-dir\") pod \"premium-w2-ba1918-kserve-85864b4646-nxprb\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") " pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:43.204155 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.204121    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-home\") pod \"premium-w2-ba1918-kserve-85864b4646-nxprb\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") " pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:43.206070 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.206044    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-dshm\") pod \"premium-w2-ba1918-kserve-85864b4646-nxprb\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") " pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:43.206617 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.206572    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-tls-certs\") pod \"premium-w2-ba1918-kserve-85864b4646-nxprb\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") " pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:43.258020 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.257963    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpfj6\" (UniqueName: \"kubernetes.io/projected/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-kube-api-access-mpfj6\") pod \"premium-w2-ba1918-kserve-85864b4646-nxprb\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") " pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:43.379661 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.379569    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:43.755448 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:43.755418    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"]
Nov 02 23:06:43.756102 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:06:43.756073    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa4f12b9_2c3e_46a1_9e99_d0dce3ceee8a.slice/crio-a642a18dec55066c521a7274f0125e2533dcdea153678b75568b3df0495571e9 WatchSource:0}: Error finding container a642a18dec55066c521a7274f0125e2533dcdea153678b75568b3df0495571e9: Status 404 returned error can't find the container with id a642a18dec55066c521a7274f0125e2533dcdea153678b75568b3df0495571e9
Nov 02 23:06:44.745680 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:44.745643    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb" event={"ID":"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a","Type":"ContainerStarted","Data":"b7a188c5191b2db40a987d9ee8261284ecd1cd8b212f142c29a09c67868cc665"}
Nov 02 23:06:44.746114 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:44.745694    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb" event={"ID":"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a","Type":"ContainerStarted","Data":"a642a18dec55066c521a7274f0125e2533dcdea153678b75568b3df0495571e9"}
Nov 02 23:06:44.857001 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:44.856928    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb" podStartSLOduration=1.856913646 podStartE2EDuration="1.856913646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:06:44.854912415 +0000 UTC m=+844.423267204" watchObservedRunningTime="2026-11-02 23:06:44.856913646 +0000 UTC m=+844.425268432"
Nov 02 23:06:45.380515 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:45.380474    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:45.648914 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:45.648827    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:06:50.160438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:50.160353    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:06:53.380036 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:53.379971    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:53.398185 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:53.398155    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:53.829085 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:53.829055    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:06:54.751224 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:54.751183    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:06:59.240005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:06:59.239955    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:07:03.834312 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:07:03.834275    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:07:07.697647 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:07:07.697612    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:07:08.352061 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:07:08.352024    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:07:11.750904 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:07:11.750859    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:07:12.950642 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:07:12.950606    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:07:17.547928 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:07:17.547893    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:07:19.492590 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:07:19.492551    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:07:22.159947 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:07:22.159916    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:07:25.779083 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:07:25.779024    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:07:26.754791 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:07:26.754757    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:07:31.438923 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:07:31.438883    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:07:36.050085 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:07:36.050044    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:07:40.584663 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:07:40.584620    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:07:41.750621 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:07:41.750584    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:07:45.045729 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:07:45.045689    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:07:49.663428 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:07:49.663372    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:07:54.398036 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:07:54.397999    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:07:58.972184 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:07:58.972141    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:03.479057 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:03.478954    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:03.699395 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:03.699355    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:03.857081 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:03.857046    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:04.019570 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:04.019539    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:08.035994 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:08.035943    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:08.831080 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:08.831047    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:11.749925 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:08:11.749884    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:08:12.726553 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:12.726521    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:17.283781 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:17.283741    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:21.305721 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:21.305679    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:21.982855 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:21.982817    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:26.091890 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:26.091801    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:26.543710 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:26.543673    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:29.893663 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:29.893630    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"]
Nov 02 23:08:29.897564 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:29.897538    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"
Nov 02 23:08:29.948642 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:29.948600    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"]
Nov 02 23:08:29.971308 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:29.971261    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-routing-overlay-volume\") pod \"payload-processing-external-model-6b9d985949-l9q7s\" (UID: \"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8\") " pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"
Nov 02 23:08:29.971483 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:29.971315    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c959s\" (UniqueName: \"kubernetes.io/projected/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-kube-api-access-c959s\") pod \"payload-processing-external-model-6b9d985949-l9q7s\" (UID: \"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8\") " pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"
Nov 02 23:08:29.971534 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:29.971476    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-provider-credentials\") pod \"payload-processing-external-model-6b9d985949-l9q7s\" (UID: \"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8\") " pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"
Nov 02 23:08:29.971572 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:29.971531    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-plugins-config-volume\") pod \"payload-processing-external-model-6b9d985949-l9q7s\" (UID: \"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8\") " pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"
Nov 02 23:08:29.992805 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:29.992773    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 23:08:29.992805 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:29.992792    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ef36b3c11cbb\""
Nov 02 23:08:29.992805 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:29.992776    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 23:08:29.993079 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:29.992849    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 23:08:29.995198 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:29.995183    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-d5hk8\""
Nov 02 23:08:29.995261 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:29.995207    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 23:08:30.073324 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.073282    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-plugins-config-volume\") pod \"payload-processing-external-model-6b9d985949-l9q7s\" (UID: \"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8\") " pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"
Nov 02 23:08:30.073528 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.073376    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-routing-overlay-volume\") pod \"payload-processing-external-model-6b9d985949-l9q7s\" (UID: \"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8\") " pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"
Nov 02 23:08:30.073528 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.073399    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c959s\" (UniqueName: \"kubernetes.io/projected/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-kube-api-access-c959s\") pod \"payload-processing-external-model-6b9d985949-l9q7s\" (UID: \"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8\") " pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"
Nov 02 23:08:30.073528 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.073469    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-provider-credentials\") pod \"payload-processing-external-model-6b9d985949-l9q7s\" (UID: \"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8\") " pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"
Nov 02 23:08:30.074071 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.074042    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-routing-overlay-volume\") pod \"payload-processing-external-model-6b9d985949-l9q7s\" (UID: \"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8\") " pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"
Nov 02 23:08:30.074205 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.074156    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-plugins-config-volume\") pod \"payload-processing-external-model-6b9d985949-l9q7s\" (UID: \"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8\") " pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"
Nov 02 23:08:30.076069 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.076045    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-provider-credentials\") pod \"payload-processing-external-model-6b9d985949-l9q7s\" (UID: \"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8\") " pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"
Nov 02 23:08:30.127800 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.127745    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c959s\" (UniqueName: \"kubernetes.io/projected/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-kube-api-access-c959s\") pod \"payload-processing-external-model-6b9d985949-l9q7s\" (UID: \"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8\") " pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"
Nov 02 23:08:30.211808 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.211716    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"
Nov 02 23:08:30.489538 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.489439    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"]
Nov 02 23:08:30.494416 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.494398    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.579827 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.579790    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.580012 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.579867    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.580012 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.579990    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.580149 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.580063    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.580346 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.580299    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.580346 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.580335    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.580482 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.580421    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.580482 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.580473    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.580577 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.580500    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcl92\" (UniqueName: \"kubernetes.io/projected/4238eef4-75f3-4b97-b84e-dcd47d4931c0-kube-api-access-wcl92\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.595781 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.595746    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"]
Nov 02 23:08:30.598654 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:08:30.597939    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e918f14_2ac1_44b7_ab9d_dccba95ba8b8.slice/crio-9cf4ee118ea84f2b40e4dd24842a3fd68d286bfdfc434b8ccbb6a13d02209815 WatchSource:0}: Error finding container 9cf4ee118ea84f2b40e4dd24842a3fd68d286bfdfc434b8ccbb6a13d02209815: Status 404 returned error can't find the container with id 9cf4ee118ea84f2b40e4dd24842a3fd68d286bfdfc434b8ccbb6a13d02209815
Nov 02 23:08:30.681996 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.681951    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.682164 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.682078    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.682224 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.682179    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.682310 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.682281    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.682414 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.682330    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.682414 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.682370    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.682578 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.682426    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.682578 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.682478    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcl92\" (UniqueName: \"kubernetes.io/projected/4238eef4-75f3-4b97-b84e-dcd47d4931c0-kube-api-access-wcl92\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.682749 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.682578    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.682749 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.682592    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.682749 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.682656    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.682749 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.682740    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.683253 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.683219    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.684720 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.684611    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.685866 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.685842    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.686052 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.685916    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.688304 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.688282    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"]
Nov 02 23:08:30.736857 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.736828    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.737061 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.737001    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcl92\" (UniqueName: \"kubernetes.io/projected/4238eef4-75f3-4b97-b84e-dcd47d4931c0-kube-api-access-wcl92\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-v4j5b\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:30.808403 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:30.808363    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:31.160128 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:31.160043    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:31.233121 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:08:31.233089    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4238eef4_75f3_4b97_b84e_dcd47d4931c0.slice/crio-8b50975328345d0b461a097890cbeb04c9953383036a808ab09155239f0f930c WatchSource:0}: Error finding container 8b50975328345d0b461a097890cbeb04c9953383036a808ab09155239f0f930c: Status 404 returned error can't find the container with id 8b50975328345d0b461a097890cbeb04c9953383036a808ab09155239f0f930c
Nov 02 23:08:31.233260 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:31.233166    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"]
Nov 02 23:08:31.235774 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:31.235744    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:08:31.235864 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:31.235808    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:08:31.235864 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:31.235837    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:08:31.415070 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:31.411955    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s" event={"ID":"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8","Type":"ContainerStarted","Data":"b91d8d85961309d59605327676936d0bdd4de5f611e227da64c19d93b22c7e9d"}
Nov 02 23:08:31.415070 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:31.412154    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"
Nov 02 23:08:31.415070 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:31.412192    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s" event={"ID":"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8","Type":"ContainerStarted","Data":"9cf4ee118ea84f2b40e4dd24842a3fd68d286bfdfc434b8ccbb6a13d02209815"}
Nov 02 23:08:31.415070 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:31.414009    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b" event={"ID":"4238eef4-75f3-4b97-b84e-dcd47d4931c0","Type":"ContainerStarted","Data":"467675910c95bbcafc4b723a4411c2aecbd10e4c5f55dc390052df3743194133"}
Nov 02 23:08:31.415070 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:31.414060    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b" event={"ID":"4238eef4-75f3-4b97-b84e-dcd47d4931c0","Type":"ContainerStarted","Data":"8b50975328345d0b461a097890cbeb04c9953383036a808ab09155239f0f930c"}
Nov 02 23:08:31.521805 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:31.521741    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s" podStartSLOduration=2.5217265319999997 podStartE2EDuration="2.521726532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:08:31.520632132 +0000 UTC m=+951.088986920" watchObservedRunningTime="2026-11-02 23:08:31.521726532 +0000 UTC m=+951.090081319"
Nov 02 23:08:31.633181 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:31.633049    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b" podStartSLOduration=1.633033355 podStartE2EDuration="1.633033355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:08:31.631632702 +0000 UTC m=+951.199987503" watchObservedRunningTime="2026-11-02 23:08:31.633033355 +0000 UTC m=+951.201388141"
Nov 02 23:08:31.808804 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:31.808766    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:32.810663 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:32.810623    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 23:08:32.811071 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:32.810690    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b" podUID="4238eef4-75f3-4b97-b84e-dcd47d4931c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 23:08:33.809285 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:33.809251    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 23:08:33.809496 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:33.809308    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b" podUID="4238eef4-75f3-4b97-b84e-dcd47d4931c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 23:08:34.809306 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:34.809267    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 23:08:34.810196 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:34.809339    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b" podUID="4238eef4-75f3-4b97-b84e-dcd47d4931c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 23:08:35.809080 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:35.809037    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 23:08:35.809301 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:35.809102    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b" podUID="4238eef4-75f3-4b97-b84e-dcd47d4931c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 23:08:35.833864 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:35.833807    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:36.809343 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:36.809295    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 23:08:36.809543 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:36.809354    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b" podUID="4238eef4-75f3-4b97-b84e-dcd47d4931c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 23:08:37.809715 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:37.809675    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 23:08:37.810380 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:37.810342    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b" podUID="4238eef4-75f3-4b97-b84e-dcd47d4931c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 23:08:38.809027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:38.808967    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 02 23:08:38.809236 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:38.809138    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b" podUID="4238eef4-75f3-4b97-b84e-dcd47d4931c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 02 23:08:40.151687 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:40.151647    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:40.152263 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:40.152056    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:40.153057 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:40.153032    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:08:40.780947 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:40.780905    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:41.749889 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:08:41.749850    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:08:42.423024 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:42.422971    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s" podUID="2e918f14-2ac1-44b7-ab9d-dccba95ba8b8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:08:45.283778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.283741    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:45.449465 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.449433    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:45.592356 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.592274    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"]
Nov 02 23:08:45.597318 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.597297    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.684563 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.684523    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.684795 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.684583    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.684795 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.684655    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.684795 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.684718    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da47e811-c010-4279-aa35-82c34f9b00cb-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.684972 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.684801    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da47e811-c010-4279-aa35-82c34f9b00cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.684972 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.684825    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da47e811-c010-4279-aa35-82c34f9b00cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.684972 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.684849    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn4dl\" (UniqueName: \"kubernetes.io/projected/da47e811-c010-4279-aa35-82c34f9b00cb-kube-api-access-xn4dl\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.684972 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.684919    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.685196 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.685016    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.697927 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.697893    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"]
Nov 02 23:08:45.785648 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.785609    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da47e811-c010-4279-aa35-82c34f9b00cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.785648 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.785648    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da47e811-c010-4279-aa35-82c34f9b00cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.785906 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.785674    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn4dl\" (UniqueName: \"kubernetes.io/projected/da47e811-c010-4279-aa35-82c34f9b00cb-kube-api-access-xn4dl\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.785906 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.785735    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.785906 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.785797    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.785906 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.785829    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.785906 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.785862    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.786195 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.785933    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.786195 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.786036    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da47e811-c010-4279-aa35-82c34f9b00cb-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.786362 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.786271    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.786505 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.786481    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da47e811-c010-4279-aa35-82c34f9b00cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.786666 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.786574    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.786666 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.786590    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.786841 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.786705    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.788276 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.788260    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.788396 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.788374    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da47e811-c010-4279-aa35-82c34f9b00cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.839948 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.839909    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da47e811-c010-4279-aa35-82c34f9b00cb-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.840157 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.839964    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn4dl\" (UniqueName: \"kubernetes.io/projected/da47e811-c010-4279-aa35-82c34f9b00cb-kube-api-access-xn4dl\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9cb6n\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:45.910898 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:45.910807    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:46.325509 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:46.325483    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"]
Nov 02 23:08:46.325867 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:08:46.325840    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda47e811_c010_4279_aa35_82c34f9b00cb.slice/crio-f0540524334e79cfe7977fe560086f9cd215c6f054b4781fc9616b85883788a3 WatchSource:0}: Error finding container f0540524334e79cfe7977fe560086f9cd215c6f054b4781fc9616b85883788a3: Status 404 returned error can't find the container with id f0540524334e79cfe7977fe560086f9cd215c6f054b4781fc9616b85883788a3
Nov 02 23:08:46.328289 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:46.328254    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:08:46.328397 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:46.328322    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:08:46.328397 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:46.328353    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:08:46.505784 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:46.505746    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n" event={"ID":"da47e811-c010-4279-aa35-82c34f9b00cb","Type":"ContainerStarted","Data":"a71f39b826cf260dd990c1451b725121870141f27a599340c8f26fb6bfdf03de"}
Nov 02 23:08:46.505989 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:46.505807    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n" event={"ID":"da47e811-c010-4279-aa35-82c34f9b00cb","Type":"ContainerStarted","Data":"f0540524334e79cfe7977fe560086f9cd215c6f054b4781fc9616b85883788a3"}
Nov 02 23:08:46.619694 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:46.619591    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n" podStartSLOduration=1.619576726 podStartE2EDuration="1.619576726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:08:46.616464599 +0000 UTC m=+966.184819386" watchObservedRunningTime="2026-11-02 23:08:46.619576726 +0000 UTC m=+966.187931512"
Nov 02 23:08:46.911115 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:46.911011    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:47.913563 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:47.913529    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 23:08:47.913964 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:47.913584    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n" podUID="da47e811-c010-4279-aa35-82c34f9b00cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 23:08:48.911970 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:48.911930    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 23:08:48.912214 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:48.912023    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n" podUID="da47e811-c010-4279-aa35-82c34f9b00cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 23:08:49.848350 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:49.848314    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:49.912130 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:49.912093    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 23:08:49.912335 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:49.912162    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n" podUID="da47e811-c010-4279-aa35-82c34f9b00cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 23:08:50.020067 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.020028    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"]
Nov 02 23:08:50.021430 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.021400    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s" podUID="2e918f14-2ac1-44b7-ab9d-dccba95ba8b8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:08:50.082371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.082336    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"]
Nov 02 23:08:50.087114 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.087090    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"
Nov 02 23:08:50.129174 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.129089    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"]
Nov 02 23:08:50.139274 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.137726    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4e90176b6698\""
Nov 02 23:08:50.141663 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.140501    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e71251f9-1dd4-4898-a61c-77ce47523239-provider-credentials\") pod \"payload-processing-external-model-6bcb8dc99c-qfffk\" (UID: \"e71251f9-1dd4-4898-a61c-77ce47523239\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"
Nov 02 23:08:50.141663 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.140572    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e71251f9-1dd4-4898-a61c-77ce47523239-routing-overlay-volume\") pod \"payload-processing-external-model-6bcb8dc99c-qfffk\" (UID: \"e71251f9-1dd4-4898-a61c-77ce47523239\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"
Nov 02 23:08:50.141663 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.140675    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71251f9-1dd4-4898-a61c-77ce47523239-plugins-config-volume\") pod \"payload-processing-external-model-6bcb8dc99c-qfffk\" (UID: \"e71251f9-1dd4-4898-a61c-77ce47523239\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"
Nov 02 23:08:50.141663 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.140732    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnmpc\" (UniqueName: \"kubernetes.io/projected/e71251f9-1dd4-4898-a61c-77ce47523239-kube-api-access-xnmpc\") pod \"payload-processing-external-model-6bcb8dc99c-qfffk\" (UID: \"e71251f9-1dd4-4898-a61c-77ce47523239\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"
Nov 02 23:08:50.241703 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.241648    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e71251f9-1dd4-4898-a61c-77ce47523239-provider-credentials\") pod \"payload-processing-external-model-6bcb8dc99c-qfffk\" (UID: \"e71251f9-1dd4-4898-a61c-77ce47523239\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"
Nov 02 23:08:50.241703 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.241718    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e71251f9-1dd4-4898-a61c-77ce47523239-routing-overlay-volume\") pod \"payload-processing-external-model-6bcb8dc99c-qfffk\" (UID: \"e71251f9-1dd4-4898-a61c-77ce47523239\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"
Nov 02 23:08:50.241935 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.241789    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71251f9-1dd4-4898-a61c-77ce47523239-plugins-config-volume\") pod \"payload-processing-external-model-6bcb8dc99c-qfffk\" (UID: \"e71251f9-1dd4-4898-a61c-77ce47523239\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"
Nov 02 23:08:50.241935 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.241842    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnmpc\" (UniqueName: \"kubernetes.io/projected/e71251f9-1dd4-4898-a61c-77ce47523239-kube-api-access-xnmpc\") pod \"payload-processing-external-model-6bcb8dc99c-qfffk\" (UID: \"e71251f9-1dd4-4898-a61c-77ce47523239\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"
Nov 02 23:08:50.242516 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.242493    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e71251f9-1dd4-4898-a61c-77ce47523239-routing-overlay-volume\") pod \"payload-processing-external-model-6bcb8dc99c-qfffk\" (UID: \"e71251f9-1dd4-4898-a61c-77ce47523239\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"
Nov 02 23:08:50.242661 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.242640    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71251f9-1dd4-4898-a61c-77ce47523239-plugins-config-volume\") pod \"payload-processing-external-model-6bcb8dc99c-qfffk\" (UID: \"e71251f9-1dd4-4898-a61c-77ce47523239\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"
Nov 02 23:08:50.244405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.244380    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e71251f9-1dd4-4898-a61c-77ce47523239-provider-credentials\") pod \"payload-processing-external-model-6bcb8dc99c-qfffk\" (UID: \"e71251f9-1dd4-4898-a61c-77ce47523239\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"
Nov 02 23:08:50.296833 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.296799    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnmpc\" (UniqueName: \"kubernetes.io/projected/e71251f9-1dd4-4898-a61c-77ce47523239-kube-api-access-xnmpc\") pod \"payload-processing-external-model-6bcb8dc99c-qfffk\" (UID: \"e71251f9-1dd4-4898-a61c-77ce47523239\") " pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"
Nov 02 23:08:50.402167 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.402075    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"
Nov 02 23:08:50.799222 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.799196    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"]
Nov 02 23:08:50.800209 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:08:50.800183    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71251f9_1dd4_4898_a61c_77ce47523239.slice/crio-d81de448c7764e0992e0fad4f480c36886479f2ffbdbfd597746dad73c130c39 WatchSource:0}: Error finding container d81de448c7764e0992e0fad4f480c36886479f2ffbdbfd597746dad73c130c39: Status 404 returned error can't find the container with id d81de448c7764e0992e0fad4f480c36886479f2ffbdbfd597746dad73c130c39
Nov 02 23:08:50.911339 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.911301    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 23:08:50.911736 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:50.911361    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n" podUID="da47e811-c010-4279-aa35-82c34f9b00cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 23:08:51.539428 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:51.539384    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk" event={"ID":"e71251f9-1dd4-4898-a61c-77ce47523239","Type":"ContainerStarted","Data":"a16e65804301c40260e78467434b9e5d36f1d3b7bf19734ac52af70e8bdc83bf"}
Nov 02 23:08:51.539647 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:51.539444    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk" event={"ID":"e71251f9-1dd4-4898-a61c-77ce47523239","Type":"ContainerStarted","Data":"d81de448c7764e0992e0fad4f480c36886479f2ffbdbfd597746dad73c130c39"}
Nov 02 23:08:51.539647 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:51.539465    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"
Nov 02 23:08:51.652464 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:51.652399    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk" podStartSLOduration=1.65237883 podStartE2EDuration="1.65237883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:08:51.649011446 +0000 UTC m=+971.217366234" watchObservedRunningTime="2026-11-02 23:08:51.65237883 +0000 UTC m=+971.220733619"
Nov 02 23:08:51.911404 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:51.911321    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 23:08:51.911404 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:51.911374    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n" podUID="da47e811-c010-4279-aa35-82c34f9b00cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 23:08:52.573281 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:52.573243    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:52.912271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:52.912173    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 23:08:52.912271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:52.912231    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n" podUID="da47e811-c010-4279-aa35-82c34f9b00cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 23:08:53.912233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:53.912194    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 02 23:08:53.912419 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:53.912264    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n" podUID="da47e811-c010-4279-aa35-82c34f9b00cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 02 23:08:54.450681 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:54.450647    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:08:55.020543 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.020481    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s" podUID="2e918f14-2ac1-44b7-ab9d-dccba95ba8b8" containerName="payload-processing" containerID="cri-o://b91d8d85961309d59605327676936d0bdd4de5f611e227da64c19d93b22c7e9d" gracePeriod=25
Nov 02 23:08:55.225419 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.225387    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:55.226605 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.226287    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:55.229258 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.229235    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:08:55.295738 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.295712    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"
Nov 02 23:08:55.425183 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.425147    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-routing-overlay-volume\") pod \"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8\" (UID: \"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8\") "
Nov 02 23:08:55.425397 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.425198    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-kube-api-access-c959s\" (UniqueName: \"kubernetes.io/projected/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-kube-api-access-c959s\") pod \"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8\" (UID: \"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8\") "
Nov 02 23:08:55.425397 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.425268    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-provider-credentials\") pod \"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8\" (UID: \"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8\") "
Nov 02 23:08:55.425516 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.425425    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-plugins-config-volume\") pod \"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8\" (UID: \"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8\") "
Nov 02 23:08:55.425565 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.425529    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-routing-overlay-volume" pod "2e918f14-2ac1-44b7-ab9d-dccba95ba8b8" (UID: "2e918f14-2ac1-44b7-ab9d-dccba95ba8b8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:08:55.425827 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.425791    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-plugins-config-volume" pod "2e918f14-2ac1-44b7-ab9d-dccba95ba8b8" (UID: "2e918f14-2ac1-44b7-ab9d-dccba95ba8b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:08:55.425945 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.425870    4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-routing-overlay-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:08:55.428297 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.428263    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-provider-credentials" pod "2e918f14-2ac1-44b7-ab9d-dccba95ba8b8" (UID: "2e918f14-2ac1-44b7-ab9d-dccba95ba8b8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:08:55.428428 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.428369    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-kube-api-access-c959s" pod "2e918f14-2ac1-44b7-ab9d-dccba95ba8b8" (UID: "2e918f14-2ac1-44b7-ab9d-dccba95ba8b8"). InnerVolumeSpecName "kube-api-access-c959s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:08:55.527384 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.527283    4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-provider-credentials\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:08:55.527384 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.527326    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:08:55.527384 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.527344    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c959s\" (UniqueName: \"kubernetes.io/projected/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8-kube-api-access-c959s\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:08:55.566857 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.566821    4294 generic.go:356] "Generic (PLEG): container finished" podID="2e918f14-2ac1-44b7-ab9d-dccba95ba8b8" containerID="b91d8d85961309d59605327676936d0bdd4de5f611e227da64c19d93b22c7e9d" exitCode=0
Nov 02 23:08:55.567045 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.566869    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"
Nov 02 23:08:55.567045 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.566900    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s" event={"ID":"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8","Type":"ContainerDied","Data":"b91d8d85961309d59605327676936d0bdd4de5f611e227da64c19d93b22c7e9d"}
Nov 02 23:08:55.567045 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.566954    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s" event={"ID":"2e918f14-2ac1-44b7-ab9d-dccba95ba8b8","Type":"ContainerDied","Data":"9cf4ee118ea84f2b40e4dd24842a3fd68d286bfdfc434b8ccbb6a13d02209815"}
Nov 02 23:08:55.567045 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.567001    4294 scope.go:122] "RemoveContainer" containerID="b91d8d85961309d59605327676936d0bdd4de5f611e227da64c19d93b22c7e9d"
Nov 02 23:08:55.576732 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.576711    4294 scope.go:122] "RemoveContainer" containerID="b91d8d85961309d59605327676936d0bdd4de5f611e227da64c19d93b22c7e9d"
Nov 02 23:08:55.577111 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:08:55.577084    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b91d8d85961309d59605327676936d0bdd4de5f611e227da64c19d93b22c7e9d\": container with ID starting with b91d8d85961309d59605327676936d0bdd4de5f611e227da64c19d93b22c7e9d not found: ID does not exist" containerID="b91d8d85961309d59605327676936d0bdd4de5f611e227da64c19d93b22c7e9d"
Nov 02 23:08:55.577212 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.577117    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b91d8d85961309d59605327676936d0bdd4de5f611e227da64c19d93b22c7e9d"} err="failed to get container status \"b91d8d85961309d59605327676936d0bdd4de5f611e227da64c19d93b22c7e9d\": rpc error: code = NotFound desc = could not find container \"b91d8d85961309d59605327676936d0bdd4de5f611e227da64c19d93b22c7e9d\": container with ID starting with b91d8d85961309d59605327676936d0bdd4de5f611e227da64c19d93b22c7e9d not found: ID does not exist"
Nov 02 23:08:55.722728 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.722694    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"]
Nov 02 23:08:55.724011 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:55.723991    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b9d985949-l9q7s"]
Nov 02 23:08:56.547877 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:56.547845    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e918f14-2ac1-44b7-ab9d-dccba95ba8b8" path="/var/lib/kubelet/pods/2e918f14-2ac1-44b7-ab9d-dccba95ba8b8/volumes"
Nov 02 23:08:59.051381 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:08:59.051346    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:09:02.550352 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:02.550302    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk" podUID="e71251f9-1dd4-4898-a61c-77ce47523239" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:09:04.258840 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:04.258803    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:09:08.847813 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:08.847774    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"]
Nov 02 23:09:08.848353 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:08.848338    4294 cpu_manager.go:414] "containerMap: removing container" podUID="2e918f14-2ac1-44b7-ab9d-dccba95ba8b8" containerName="payload-processing"
Nov 02 23:09:08.848397 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:08.848355    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e918f14-2ac1-44b7-ab9d-dccba95ba8b8" containerName="payload-processing"
Nov 02 23:09:08.848474 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:08.848464    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e918f14-2ac1-44b7-ab9d-dccba95ba8b8" containerName="payload-processing"
Nov 02 23:09:08.851990 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:08.851963    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:08.897863 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:08.897824    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:09:08.903760 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:08.903724    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-dshm\") pod \"distinct-w1-3d8404-kserve-549fc4f59f-wfmxh\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:08.903966 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:08.903859    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7wbj\" (UniqueName: \"kubernetes.io/projected/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-kube-api-access-w7wbj\") pod \"distinct-w1-3d8404-kserve-549fc4f59f-wfmxh\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:08.903966 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:08.903933    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-model-cache\") pod \"distinct-w1-3d8404-kserve-549fc4f59f-wfmxh\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:08.904154 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:08.904128    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-home\") pod \"distinct-w1-3d8404-kserve-549fc4f59f-wfmxh\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:08.904229 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:08.904162    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-tmp-dir\") pod \"distinct-w1-3d8404-kserve-549fc4f59f-wfmxh\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:08.904287 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:08.904264    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-tls-certs\") pod \"distinct-w1-3d8404-kserve-549fc4f59f-wfmxh\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:08.907917 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:08.907883    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"]
Nov 02 23:09:08.948571 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:08.948540    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3d8404\"/\"distinct-w1-3d8404-kserve-self-signed-certs\""
Nov 02 23:09:09.007397 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:09.007353    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7wbj\" (UniqueName: \"kubernetes.io/projected/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-kube-api-access-w7wbj\") pod \"distinct-w1-3d8404-kserve-549fc4f59f-wfmxh\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:09.007688 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:09.007664    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-model-cache\") pod \"distinct-w1-3d8404-kserve-549fc4f59f-wfmxh\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:09.007794 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:09.007780    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-home\") pod \"distinct-w1-3d8404-kserve-549fc4f59f-wfmxh\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:09.008017 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:09.007969    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-tmp-dir\") pod \"distinct-w1-3d8404-kserve-549fc4f59f-wfmxh\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:09.008161 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:09.008034    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-tls-certs\") pod \"distinct-w1-3d8404-kserve-549fc4f59f-wfmxh\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:09.008303 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:09.008278    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-model-cache\") pod \"distinct-w1-3d8404-kserve-549fc4f59f-wfmxh\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:09.008367 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:09.008324    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-home\") pod \"distinct-w1-3d8404-kserve-549fc4f59f-wfmxh\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:09.008708 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:09.008659    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-tmp-dir\") pod \"distinct-w1-3d8404-kserve-549fc4f59f-wfmxh\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:09.010092 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:09.010053    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-dshm\") pod \"distinct-w1-3d8404-kserve-549fc4f59f-wfmxh\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:09.011626 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:09.011604    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-tls-certs\") pod \"distinct-w1-3d8404-kserve-549fc4f59f-wfmxh\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:09.015683 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:09.015655    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-dshm\") pod \"distinct-w1-3d8404-kserve-549fc4f59f-wfmxh\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:09.062323 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:09.062289    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7wbj\" (UniqueName: \"kubernetes.io/projected/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-kube-api-access-w7wbj\") pod \"distinct-w1-3d8404-kserve-549fc4f59f-wfmxh\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") " pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:09.065850 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:09.065819    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:09:09.164254 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:09.164164    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:09.556716 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:09.556689    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"]
Nov 02 23:09:09.557298 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:09:09.557267    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode530f6fc_adcd_402a_b7e8_64d7ee34a7a6.slice/crio-da59dfdc50565b54aec988cbd46cc38a5d3fb2351d6825a100d97e8a5523b491 WatchSource:0}: Error finding container da59dfdc50565b54aec988cbd46cc38a5d3fb2351d6825a100d97e8a5523b491: Status 404 returned error can't find the container with id da59dfdc50565b54aec988cbd46cc38a5d3fb2351d6825a100d97e8a5523b491
Nov 02 23:09:09.673112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:09.673073    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh" event={"ID":"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6","Type":"ContainerStarted","Data":"78183a5d4d1d1598cca5c36c80f77997391b94d96e0cd51818859b4aca2f60fe"}
Nov 02 23:09:09.673112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:09.673118    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh" event={"ID":"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6","Type":"ContainerStarted","Data":"da59dfdc50565b54aec988cbd46cc38a5d3fb2351d6825a100d97e8a5523b491"}
Nov 02 23:09:09.785608 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:09.785548    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh" podStartSLOduration=1.785532216 podStartE2EDuration="1.785532216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:09.785222387 +0000 UTC m=+989.353577174" watchObservedRunningTime="2026-11-02 23:09:09.785532216 +0000 UTC m=+989.353887002"
Nov 02 23:09:10.164880 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:10.164841    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:11.750614 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:09:11.750575    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:09:12.548114 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:12.548073    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk" podUID="e71251f9-1dd4-4898-a61c-77ce47523239" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:09:13.465825 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:13.465788    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:09:18.056274 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:18.056237    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:09:19.165322 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:19.165287    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:19.177818 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:19.177788    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:19.740471 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:19.740441    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:09:22.548624 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:22.548576    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk" podUID="e71251f9-1dd4-4898-a61c-77ce47523239" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:09:22.640954 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:22.640920    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:09:26.440493 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:26.440456    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:09:27.140859 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:27.140811    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:09:28.721318 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.721284    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"]
Nov 02 23:09:28.725160 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.725129    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:28.822755 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.822718    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fb6e1d\"/\"e2e-conflict-b-11bf0f82-kserve-self-signed-certs\""
Nov 02 23:09:28.824269 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.824243    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"]
Nov 02 23:09:28.834632 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.834589    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:09:28.850547 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.850512    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-tmp-dir\") pod \"e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:28.850547 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.850562    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-home\") pod \"e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:28.850820 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.850743    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhvpm\" (UniqueName: \"kubernetes.io/projected/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-kube-api-access-hhvpm\") pod \"e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:28.850820 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.850807    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-tls-certs\") pod \"e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:28.850899 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.850878    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-dshm\") pod \"e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:28.850937 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.850907    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-model-cache\") pod \"e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:28.952669 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.952636    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-tmp-dir\") pod \"e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:28.952851 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.952677    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-home\") pod \"e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:28.952851 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.952769    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhvpm\" (UniqueName: \"kubernetes.io/projected/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-kube-api-access-hhvpm\") pod \"e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:28.952851 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.952818    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-tls-certs\") pod \"e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:28.953065 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.952853    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-dshm\") pod \"e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:28.953065 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.952878    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-model-cache\") pod \"e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:28.953203 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.953170    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-tmp-dir\") pod \"e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:28.953277 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.953226    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-home\") pod \"e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:28.953277 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.953245    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-model-cache\") pod \"e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:28.955208 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.955175    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-dshm\") pod \"e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:28.955416 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:28.955400    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-tls-certs\") pod \"e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:29.007777 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:29.007740    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhvpm\" (UniqueName: \"kubernetes.io/projected/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-kube-api-access-hhvpm\") pod \"e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") " pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:29.046678 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:29.046639    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:29.430244 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:29.430216    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"]
Nov 02 23:09:29.431201 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:09:29.431166    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ed7da65_a03d_45d7_aaf0_c79b3920b3ec.slice/crio-d5f26b8ff8ce05e301be42bcbccef04e867f2d6d8fdbbe0ad275225cd7849bde WatchSource:0}: Error finding container d5f26b8ff8ce05e301be42bcbccef04e867f2d6d8fdbbe0ad275225cd7849bde: Status 404 returned error can't find the container with id d5f26b8ff8ce05e301be42bcbccef04e867f2d6d8fdbbe0ad275225cd7849bde
Nov 02 23:09:29.792906 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:29.792865    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c" event={"ID":"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec","Type":"ContainerStarted","Data":"e535903dedc5361c0c79e5a4fbeee77bdd68406ab136ed7c72edd6faff8ac6a2"}
Nov 02 23:09:29.793407 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:29.792918    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c" event={"ID":"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec","Type":"ContainerStarted","Data":"d5f26b8ff8ce05e301be42bcbccef04e867f2d6d8fdbbe0ad275225cd7849bde"}
Nov 02 23:09:29.903618 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:29.903557    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c" podStartSLOduration=1.903541286 podStartE2EDuration="1.903541286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:09:29.901771299 +0000 UTC m=+1009.470126087" watchObservedRunningTime="2026-11-02 23:09:29.903541286 +0000 UTC m=+1009.471896077"
Nov 02 23:09:30.047178 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:30.047083    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:31.798839 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:31.798808    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:09:32.548633 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:32.548594    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk" podUID="e71251f9-1dd4-4898-a61c-77ce47523239" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:09:36.346323 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:36.346282    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:09:39.047649 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:39.047603    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:39.060794 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:39.060768    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:39.880234 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:39.880206    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:09:40.848633 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:40.848600    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:09:41.754587 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:09:41.754547    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:09:42.551252 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:42.551205    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk" podUID="e71251f9-1dd4-4898-a61c-77ce47523239" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:09:45.343584 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:45.343546    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:09:49.947716 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:49.947647    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:09:52.548588 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:52.548555    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk" podUID="e71251f9-1dd4-4898-a61c-77ce47523239" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:09:54.798747 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:54.798650    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:09:55.091950 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:55.091866    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:09:55.246954 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:55.246924    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:09:55.414607 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:55.414515    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:09:58.541847 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:58.541806    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk" podUID="e71251f9-1dd4-4898-a61c-77ce47523239" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:09:59.291120 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:59.291064    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"]
Nov 02 23:09:59.292387 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:59.292358    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk" podUID="e71251f9-1dd4-4898-a61c-77ce47523239" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:09:59.496258 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:09:59.496217    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:00.414155 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.414121    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"]
Nov 02 23:10:00.418162 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.418146    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.499092 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.499020    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"]
Nov 02 23:10:00.522022 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.521971    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.522210 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.522090    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.522210 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.522118    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.522291 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.522234    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52a13074-c991-4d08-9bfd-c132ffab4c90-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.522436 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.522415    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.522475 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.522451    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.522536 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.522509    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhgcx\" (UniqueName: \"kubernetes.io/projected/52a13074-c991-4d08-9bfd-c132ffab4c90-kube-api-access-xhgcx\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.522583 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.522552    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.522583 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.522571    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.623879 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.623834    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.623879 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.623884    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.624164 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.623913    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhgcx\" (UniqueName: \"kubernetes.io/projected/52a13074-c991-4d08-9bfd-c132ffab4c90-kube-api-access-xhgcx\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.624164 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.623939    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.624164 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.623963    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.624329 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.624206    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.624329 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.624269    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.624329 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.624297    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.624493 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.624392    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52a13074-c991-4d08-9bfd-c132ffab4c90-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.624493 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.624414    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.624493 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.624449    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.624743 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.624717    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.624857 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.624836    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.625201 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.625176    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52a13074-c991-4d08-9bfd-c132ffab4c90-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.626533 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.626514    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.626657 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.626639    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.678495 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.678416    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhgcx\" (UniqueName: \"kubernetes.io/projected/52a13074-c991-4d08-9bfd-c132ffab4c90-kube-api-access-xhgcx\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.678495 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.678429    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:00.731759 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:00.731723    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:01.155347 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:01.155319    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"]
Nov 02 23:10:01.156681 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:10:01.156648    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a13074_c991_4d08_9bfd_c132ffab4c90.slice/crio-19e47e8933404ba45061021cb12a1a7343af669a8c53bbb33d8569b363e31a14 WatchSource:0}: Error finding container 19e47e8933404ba45061021cb12a1a7343af669a8c53bbb33d8569b363e31a14: Status 404 returned error can't find the container with id 19e47e8933404ba45061021cb12a1a7343af669a8c53bbb33d8569b363e31a14
Nov 02 23:10:01.159092 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:01.159062    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:10:01.159180 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:01.159124    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:10:01.159180 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:01.159155    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:10:01.998312 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:01.998272    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq" event={"ID":"52a13074-c991-4d08-9bfd-c132ffab4c90","Type":"ContainerStarted","Data":"616c8c98911f49a0700c674bd8f5d70b55d3e18d508732769f818a750011cc0d"}
Nov 02 23:10:01.998706 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:01.998333    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq" event={"ID":"52a13074-c991-4d08-9bfd-c132ffab4c90","Type":"ContainerStarted","Data":"19e47e8933404ba45061021cb12a1a7343af669a8c53bbb33d8569b363e31a14"}
Nov 02 23:10:02.213350 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:02.213292    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq" podStartSLOduration=2.213277085 podStartE2EDuration="2.213277085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:02.164461812 +0000 UTC m=+1041.732816599" watchObservedRunningTime="2026-11-02 23:10:02.213277085 +0000 UTC m=+1041.781631872"
Nov 02 23:10:02.580780 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:02.580738    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:02.738008 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:02.732791    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:02.738008 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:02.734735    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 23:10:02.738008 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:02.734796    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq" podUID="52a13074-c991-4d08-9bfd-c132ffab4c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 23:10:03.732911 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:03.732669    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 23:10:03.732911 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:03.732757    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq" podUID="52a13074-c991-4d08-9bfd-c132ffab4c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 23:10:04.167706 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:04.167626    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:04.292112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:04.292057    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk" podUID="e71251f9-1dd4-4898-a61c-77ce47523239" containerName="payload-processing" containerID="cri-o://a16e65804301c40260e78467434b9e5d36f1d3b7bf19734ac52af70e8bdc83bf" gracePeriod=25
Nov 02 23:10:04.734442 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:04.734343    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 23:10:04.739872 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:04.739828    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq" podUID="52a13074-c991-4d08-9bfd-c132ffab4c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 23:10:05.268032 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:05.259573    4294 generic.go:356] "Generic (PLEG): container finished" podID="e71251f9-1dd4-4898-a61c-77ce47523239" containerID="a16e65804301c40260e78467434b9e5d36f1d3b7bf19734ac52af70e8bdc83bf" exitCode=0
Nov 02 23:10:05.268032 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:05.259766    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk" event={"ID":"e71251f9-1dd4-4898-a61c-77ce47523239","Type":"ContainerDied","Data":"a16e65804301c40260e78467434b9e5d36f1d3b7bf19734ac52af70e8bdc83bf"}
Nov 02 23:10:05.315168 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:05.315132    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"
Nov 02 23:10:05.495827 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:05.494486    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e71251f9-1dd4-4898-a61c-77ce47523239-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e71251f9-1dd4-4898-a61c-77ce47523239-routing-overlay-volume\") pod \"e71251f9-1dd4-4898-a61c-77ce47523239\" (UID: \"e71251f9-1dd4-4898-a61c-77ce47523239\") "
Nov 02 23:10:05.495827 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:05.494663    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e71251f9-1dd4-4898-a61c-77ce47523239-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71251f9-1dd4-4898-a61c-77ce47523239-plugins-config-volume\") pod \"e71251f9-1dd4-4898-a61c-77ce47523239\" (UID: \"e71251f9-1dd4-4898-a61c-77ce47523239\") "
Nov 02 23:10:05.495827 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:05.494716    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71251f9-1dd4-4898-a61c-77ce47523239-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e71251f9-1dd4-4898-a61c-77ce47523239-provider-credentials\") pod \"e71251f9-1dd4-4898-a61c-77ce47523239\" (UID: \"e71251f9-1dd4-4898-a61c-77ce47523239\") "
Nov 02 23:10:05.495827 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:05.494850    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71251f9-1dd4-4898-a61c-77ce47523239-kube-api-access-xnmpc\" (UniqueName: \"kubernetes.io/projected/e71251f9-1dd4-4898-a61c-77ce47523239-kube-api-access-xnmpc\") pod \"e71251f9-1dd4-4898-a61c-77ce47523239\" (UID: \"e71251f9-1dd4-4898-a61c-77ce47523239\") "
Nov 02 23:10:05.518509 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:05.516717    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71251f9-1dd4-4898-a61c-77ce47523239-plugins-config-volume" pod "e71251f9-1dd4-4898-a61c-77ce47523239" (UID: "e71251f9-1dd4-4898-a61c-77ce47523239"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:10:05.518509 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:05.517032    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71251f9-1dd4-4898-a61c-77ce47523239-kube-api-access-xnmpc" pod "e71251f9-1dd4-4898-a61c-77ce47523239" (UID: "e71251f9-1dd4-4898-a61c-77ce47523239"). InnerVolumeSpecName "kube-api-access-xnmpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:10:05.531028 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:05.521202    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71251f9-1dd4-4898-a61c-77ce47523239-routing-overlay-volume" pod "e71251f9-1dd4-4898-a61c-77ce47523239" (UID: "e71251f9-1dd4-4898-a61c-77ce47523239"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:10:05.555225 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:05.548772    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71251f9-1dd4-4898-a61c-77ce47523239-provider-credentials" pod "e71251f9-1dd4-4898-a61c-77ce47523239" (UID: "e71251f9-1dd4-4898-a61c-77ce47523239"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:10:05.600182 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:05.600131    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnmpc\" (UniqueName: \"kubernetes.io/projected/e71251f9-1dd4-4898-a61c-77ce47523239-kube-api-access-xnmpc\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:10:05.600472 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:05.600409    4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e71251f9-1dd4-4898-a61c-77ce47523239-routing-overlay-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:10:05.600472 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:05.600436    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71251f9-1dd4-4898-a61c-77ce47523239-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:10:05.600472 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:05.600451    4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e71251f9-1dd4-4898-a61c-77ce47523239-provider-credentials\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:10:05.735139 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:05.735102    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 23:10:05.735765 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:05.735728    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq" podUID="52a13074-c991-4d08-9bfd-c132ffab4c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 23:10:05.920108 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:05.919458    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c" podUID="3ed7da65-a03d-45d7-aaf0-c79b3920b3ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:10:06.282759 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:06.282729    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"
Nov 02 23:10:06.282960 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:06.282723    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk" event={"ID":"e71251f9-1dd4-4898-a61c-77ce47523239","Type":"ContainerDied","Data":"d81de448c7764e0992e0fad4f480c36886479f2ffbdbfd597746dad73c130c39"}
Nov 02 23:10:06.282960 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:06.282886    4294 scope.go:122] "RemoveContainer" containerID="a16e65804301c40260e78467434b9e5d36f1d3b7bf19734ac52af70e8bdc83bf"
Nov 02 23:10:06.439738 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:06.439706    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"]
Nov 02 23:10:06.442435 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:06.442414    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bcb8dc99c-qfffk"]
Nov 02 23:10:06.547600 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:06.547521    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e71251f9-1dd4-4898-a61c-77ce47523239" path="/var/lib/kubelet/pods/e71251f9-1dd4-4898-a61c-77ce47523239/volumes"
Nov 02 23:10:06.732322 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:06.732283    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 23:10:06.732505 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:06.732344    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq" podUID="52a13074-c991-4d08-9bfd-c132ffab4c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 23:10:07.732030 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:07.731993    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 23:10:07.732439 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:07.732046    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq" podUID="52a13074-c991-4d08-9bfd-c132ffab4c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 23:10:08.808487 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:08.808433    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:08.809337 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:08.809313    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:08.810968 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:08.810939    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:10:08.943199 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:08.943124    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:10.654858 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:10.654824    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:11.749426 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:10:11.749388    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:10:13.659395 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:13.659352    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:18.260770 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:18.260728    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:23.150686 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:23.150647    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:24.394874 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.394832    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"]
Nov 02 23:10:24.396282 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.396250    4294 cpu_manager.go:414] "containerMap: removing container" podUID="e71251f9-1dd4-4898-a61c-77ce47523239" containerName="payload-processing"
Nov 02 23:10:24.396453 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.396437    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e71251f9-1dd4-4898-a61c-77ce47523239" containerName="payload-processing"
Nov 02 23:10:24.396728 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.396710    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e71251f9-1dd4-4898-a61c-77ce47523239" containerName="payload-processing"
Nov 02 23:10:24.405353 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.405321    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:24.450272 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.450240    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"]
Nov 02 23:10:24.494939 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.494904    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-59f97a\"/\"default-dockercfg-lln4k\""
Nov 02 23:10:24.494939 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.494937    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-59f97a\"/\"openshift-service-ca.crt\""
Nov 02 23:10:24.502654 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.502628    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-59f97a\"/\"kube-root-ca.crt\""
Nov 02 23:10:24.502654 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.502627    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-59f97a\"/\"distinct-w0-59f97a-kserve-self-signed-certs\""
Nov 02 23:10:24.507698 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.507675    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:24.545013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.544969    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-home\") pod \"distinct-w0-59f97a-kserve-8f86ccf58-h76gz\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:24.545217 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.545092    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40c2149-5d44-44e5-b271-6ebd82197e22-tls-certs\") pod \"distinct-w0-59f97a-kserve-8f86ccf58-h76gz\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:24.545302 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.545248    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj5zb\" (UniqueName: \"kubernetes.io/projected/d40c2149-5d44-44e5-b271-6ebd82197e22-kube-api-access-bj5zb\") pod \"distinct-w0-59f97a-kserve-8f86ccf58-h76gz\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:24.545302 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.545287    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-dshm\") pod \"distinct-w0-59f97a-kserve-8f86ccf58-h76gz\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:24.545411 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.545385    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-model-cache\") pod \"distinct-w0-59f97a-kserve-8f86ccf58-h76gz\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:24.545465 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.545431    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-tmp-dir\") pod \"distinct-w0-59f97a-kserve-8f86ccf58-h76gz\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:24.646435 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.646332    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5zb\" (UniqueName: \"kubernetes.io/projected/d40c2149-5d44-44e5-b271-6ebd82197e22-kube-api-access-bj5zb\") pod \"distinct-w0-59f97a-kserve-8f86ccf58-h76gz\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:24.646435 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.646375    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-dshm\") pod \"distinct-w0-59f97a-kserve-8f86ccf58-h76gz\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:24.646435 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.646409    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-model-cache\") pod \"distinct-w0-59f97a-kserve-8f86ccf58-h76gz\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:24.646730 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.646457    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-tmp-dir\") pod \"distinct-w0-59f97a-kserve-8f86ccf58-h76gz\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:24.646730 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.646628    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-home\") pod \"distinct-w0-59f97a-kserve-8f86ccf58-h76gz\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:24.646730 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.646659    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40c2149-5d44-44e5-b271-6ebd82197e22-tls-certs\") pod \"distinct-w0-59f97a-kserve-8f86ccf58-h76gz\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:24.646948 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.646919    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-model-cache\") pod \"distinct-w0-59f97a-kserve-8f86ccf58-h76gz\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:24.647030 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.646950    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-tmp-dir\") pod \"distinct-w0-59f97a-kserve-8f86ccf58-h76gz\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:24.647147 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.647121    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-home\") pod \"distinct-w0-59f97a-kserve-8f86ccf58-h76gz\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:24.648810 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.648788    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-dshm\") pod \"distinct-w0-59f97a-kserve-8f86ccf58-h76gz\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:24.649104 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.649087    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40c2149-5d44-44e5-b271-6ebd82197e22-tls-certs\") pod \"distinct-w0-59f97a-kserve-8f86ccf58-h76gz\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:24.700989 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.700955    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj5zb\" (UniqueName: \"kubernetes.io/projected/d40c2149-5d44-44e5-b271-6ebd82197e22-kube-api-access-bj5zb\") pod \"distinct-w0-59f97a-kserve-8f86ccf58-h76gz\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:24.718903 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:24.718863    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:25.126433 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:25.126400    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"]
Nov 02 23:10:25.127170 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:10:25.127144    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd40c2149_5d44_44e5_b271_6ebd82197e22.slice/crio-2ac68cc06889ad718809d202994bec2cc62a6040ef76e6cefd94a27aeece314b WatchSource:0}: Error finding container 2ac68cc06889ad718809d202994bec2cc62a6040ef76e6cefd94a27aeece314b: Status 404 returned error can't find the container with id 2ac68cc06889ad718809d202994bec2cc62a6040ef76e6cefd94a27aeece314b
Nov 02 23:10:25.441583 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:25.441494    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" event={"ID":"d40c2149-5d44-44e5-b271-6ebd82197e22","Type":"ContainerStarted","Data":"1e28a71498a14d121ff7e9e919439fdd933e24883f0424eb7706ba2c79084473"}
Nov 02 23:10:25.441583 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:25.441536    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" event={"ID":"d40c2149-5d44-44e5-b271-6ebd82197e22","Type":"ContainerStarted","Data":"2ac68cc06889ad718809d202994bec2cc62a6040ef76e6cefd94a27aeece314b"}
Nov 02 23:10:25.552671 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:25.552617    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" podStartSLOduration=1.552604071 podStartE2EDuration="1.552604071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:25.550885679 +0000 UTC m=+1065.119240469" watchObservedRunningTime="2026-11-02 23:10:25.552604071 +0000 UTC m=+1065.120958858"
Nov 02 23:10:25.719140 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:25.719052    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:26.785172 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:26.785140    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:27.746052 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:27.746014    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:32.299167 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:32.299134    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:34.582814 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:34.582778    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:34.719192 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:34.719155    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:34.732814 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:34.732785    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:34.742479 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:34.742453    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:35.512695 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:35.512669    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:10:36.886520 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:36.886489    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:41.731588 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:41.731555    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:41.754903 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:10:41.754867    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:10:46.389999 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:46.389858    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:49.211488 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:49.211441    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"]
Nov 02 23:10:49.296094 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:49.296059    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:50.963634 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:50.963602    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:54.622210 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:54.622161    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:55.634898 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:55.634868    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:10:56.200600 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.200562    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"]
Nov 02 23:10:56.206078 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.206056    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.259037 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.258997    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"]
Nov 02 23:10:56.303267 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.303237    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-59f97a-openshift-default-dockercfg-r7nkq\""
Nov 02 23:10:56.309828 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.309786    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gzb6\" (UniqueName: \"kubernetes.io/projected/b007881f-8d1a-487c-ab6f-0ad740dd047a-kube-api-access-5gzb6\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.309952 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.309913    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.310014 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.309959    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.310067 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.310016    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.310067 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.310059    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.310158 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.310106    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b007881f-8d1a-487c-ab6f-0ad740dd047a-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.310199 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.310168    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.310242 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.310204    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.310296 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.310243    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.411541 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.411505    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gzb6\" (UniqueName: \"kubernetes.io/projected/b007881f-8d1a-487c-ab6f-0ad740dd047a-kube-api-access-5gzb6\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.411703 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.411575    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.411703 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.411603    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.411703 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.411627    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.411703 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.411646    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.411703 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.411670    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b007881f-8d1a-487c-ab6f-0ad740dd047a-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.411963 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.411724    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.411963 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.411778    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.411963 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.411836    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.412193 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.412164    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.412258 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.412217    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.412390 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.412363    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.412448 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.412419    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.412656 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.412638    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b007881f-8d1a-487c-ab6f-0ad740dd047a-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.413929 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.413901    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.414154 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.414138    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.466233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.466170    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gzb6\" (UniqueName: \"kubernetes.io/projected/b007881f-8d1a-487c-ab6f-0ad740dd047a-kube-api-access-5gzb6\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.466338 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.466263    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.519236 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.519197    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:56.933366 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.933332    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"]
Nov 02 23:10:56.933696 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:10:56.933417    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb007881f_8d1a_487c_ab6f_0ad740dd047a.slice/crio-37471dafc7bac8f629b84e3271b98b07214a1174637882e27267900c6d6ffa02 WatchSource:0}: Error finding container 37471dafc7bac8f629b84e3271b98b07214a1174637882e27267900c6d6ffa02: Status 404 returned error can't find the container with id 37471dafc7bac8f629b84e3271b98b07214a1174637882e27267900c6d6ffa02
Nov 02 23:10:56.935766 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.935737    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:10:56.935850 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.935803    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:10:56.935850 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:56.935832    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:10:57.627244 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:57.627207    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg" event={"ID":"b007881f-8d1a-487c-ab6f-0ad740dd047a","Type":"ContainerStarted","Data":"02a9e8323a63b9d2981e06b1c6a74041cb12c22839f9eb18785395a91bb38b98"}
Nov 02 23:10:57.627426 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:57.627270    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg" event={"ID":"b007881f-8d1a-487c-ab6f-0ad740dd047a","Type":"ContainerStarted","Data":"37471dafc7bac8f629b84e3271b98b07214a1174637882e27267900c6d6ffa02"}
Nov 02 23:10:57.740960 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:57.740910    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg" podStartSLOduration=1.740897027 podStartE2EDuration="1.740897027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:10:57.739773694 +0000 UTC m=+1097.308128483" watchObservedRunningTime="2026-11-02 23:10:57.740897027 +0000 UTC m=+1097.309251816"
Nov 02 23:10:58.520122 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:58.520089    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:10:58.521593 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:58.521568    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 23:10:58.521727 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:58.521613    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg" podUID="b007881f-8d1a-487c-ab6f-0ad740dd047a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 23:10:59.520018 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:59.519967    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 23:10:59.520283 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:10:59.520034    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg" podUID="b007881f-8d1a-487c-ab6f-0ad740dd047a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 23:11:00.247095 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:00.247040    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:11:00.520248 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:00.520172    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 23:11:00.520248 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:00.520220    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg" podUID="b007881f-8d1a-487c-ab6f-0ad740dd047a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 23:11:01.520532 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:01.520493    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 23:11:01.520944 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:01.520549    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg" podUID="b007881f-8d1a-487c-ab6f-0ad740dd047a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 23:11:02.449762 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:02.449715    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:11:02.519803 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:02.519774    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 23:11:02.519996 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:02.519835    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg" podUID="b007881f-8d1a-487c-ab6f-0ad740dd047a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 23:11:03.519947 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:03.519905    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 02 23:11:03.520363 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:03.519971    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg" podUID="b007881f-8d1a-487c-ab6f-0ad740dd047a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 02 23:11:04.231831 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.231784    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c" podUID="3ed7da65-a03d-45d7-aaf0-c79b3920b3ec" containerName="main" containerID="cri-o://e535903dedc5361c0c79e5a4fbeee77bdd68406ab136ed7c72edd6faff8ac6a2" gracePeriod=45
Nov 02 23:11:04.488795 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.488731    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:11:04.524954 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.524925    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:11:04.525391 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.525246    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:11:04.526074 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.526057    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 02 23:11:04.530376 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.530359    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-kube-api-access-hhvpm\" (UniqueName: \"kubernetes.io/projected/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-kube-api-access-hhvpm\") pod \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") "
Nov 02 23:11:04.530438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.530420    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-tls-certs\" (UniqueName: \"kubernetes.io/secret/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-tls-certs\") pod \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") "
Nov 02 23:11:04.530477 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.530451    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-home\" (UniqueName: \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-home\") pod \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") "
Nov 02 23:11:04.530533 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.530488    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-tmp-dir\") pod \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") "
Nov 02 23:11:04.530533 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.530509    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-model-cache\") pod \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") "
Nov 02 23:11:04.530634 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.530558    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-dshm\") pod \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\" (UID: \"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec\") "
Nov 02 23:11:04.530836 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.530751    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-tmp-dir" pod "3ed7da65-a03d-45d7-aaf0-c79b3920b3ec" (UID: "3ed7da65-a03d-45d7-aaf0-c79b3920b3ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:11:04.530836 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.530764    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-home" pod "3ed7da65-a03d-45d7-aaf0-c79b3920b3ec" (UID: "3ed7da65-a03d-45d7-aaf0-c79b3920b3ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:11:04.531031 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.530839    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-model-cache" pod "3ed7da65-a03d-45d7-aaf0-c79b3920b3ec" (UID: "3ed7da65-a03d-45d7-aaf0-c79b3920b3ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:11:04.531076 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.531061    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:11:04.531120 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.531080    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:11:04.531120 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.531093    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:11:04.532733 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.532711    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-kube-api-access-hhvpm" pod "3ed7da65-a03d-45d7-aaf0-c79b3920b3ec" (UID: "3ed7da65-a03d-45d7-aaf0-c79b3920b3ec"). InnerVolumeSpecName "kube-api-access-hhvpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:11:04.532818 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.532741    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-dshm" pod "3ed7da65-a03d-45d7-aaf0-c79b3920b3ec" (UID: "3ed7da65-a03d-45d7-aaf0-c79b3920b3ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:11:04.532818 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.532767    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-tls-certs" pod "3ed7da65-a03d-45d7-aaf0-c79b3920b3ec" (UID: "3ed7da65-a03d-45d7-aaf0-c79b3920b3ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:11:04.632635 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.632492    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:11:04.632635 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.632525    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:11:04.632635 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.632540    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhvpm\" (UniqueName: \"kubernetes.io/projected/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec-kube-api-access-hhvpm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:11:04.695520 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.695487    4294 generic.go:356] "Generic (PLEG): container finished" podID="3ed7da65-a03d-45d7-aaf0-c79b3920b3ec" containerID="e535903dedc5361c0c79e5a4fbeee77bdd68406ab136ed7c72edd6faff8ac6a2" exitCode=0
Nov 02 23:11:04.695720 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.695540    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"
Nov 02 23:11:04.695720 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.695572    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c" event={"ID":"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec","Type":"ContainerDied","Data":"e535903dedc5361c0c79e5a4fbeee77bdd68406ab136ed7c72edd6faff8ac6a2"}
Nov 02 23:11:04.695720 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.695611    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c" event={"ID":"3ed7da65-a03d-45d7-aaf0-c79b3920b3ec","Type":"ContainerDied","Data":"d5f26b8ff8ce05e301be42bcbccef04e867f2d6d8fdbbe0ad275225cd7849bde"}
Nov 02 23:11:04.695720 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.695628    4294 scope.go:122] "RemoveContainer" containerID="e535903dedc5361c0c79e5a4fbeee77bdd68406ab136ed7c72edd6faff8ac6a2"
Nov 02 23:11:04.707847 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.707824    4294 scope.go:122] "RemoveContainer" containerID="e535903dedc5361c0c79e5a4fbeee77bdd68406ab136ed7c72edd6faff8ac6a2"
Nov 02 23:11:04.708872 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:11:04.708790    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e535903dedc5361c0c79e5a4fbeee77bdd68406ab136ed7c72edd6faff8ac6a2\": container with ID starting with e535903dedc5361c0c79e5a4fbeee77bdd68406ab136ed7c72edd6faff8ac6a2 not found: ID does not exist" containerID="e535903dedc5361c0c79e5a4fbeee77bdd68406ab136ed7c72edd6faff8ac6a2"
Nov 02 23:11:04.708872 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.708844    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e535903dedc5361c0c79e5a4fbeee77bdd68406ab136ed7c72edd6faff8ac6a2"} err="failed to get container status \"e535903dedc5361c0c79e5a4fbeee77bdd68406ab136ed7c72edd6faff8ac6a2\": rpc error: code = NotFound desc = could not find container \"e535903dedc5361c0c79e5a4fbeee77bdd68406ab136ed7c72edd6faff8ac6a2\": container with ID starting with e535903dedc5361c0c79e5a4fbeee77bdd68406ab136ed7c72edd6faff8ac6a2 not found: ID does not exist"
Nov 02 23:11:04.816225 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.816194    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"]
Nov 02 23:11:04.823392 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.823364    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/e2e-conflict-b-11bf0f82-kserve-647bb9779-rlp6c"]
Nov 02 23:11:04.851856 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:04.851816    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:11:06.243055 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:06.243018    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:11:06.575974 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:06.575893    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed7da65-a03d-45d7-aaf0-c79b3920b3ec" path="/var/lib/kubelet/pods/3ed7da65-a03d-45d7-aaf0-c79b3920b3ec/volumes"
Nov 02 23:11:09.449836 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:09.449798    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:11:11.752326 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:11:11.752290    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:11:14.158917 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:14.158883    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:11:18.658344 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:18.658311    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:11:23.248751 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:23.248718    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:11:25.339044 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:25.339012    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:11:27.142097 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:27.142057    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:11:27.985560 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:27.985521    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:11:31.122565 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.122527    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"]
Nov 02 23:11:31.123114 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.123105    4294 cpu_manager.go:414] "containerMap: removing container" podUID="3ed7da65-a03d-45d7-aaf0-c79b3920b3ec" containerName="main"
Nov 02 23:11:31.123167 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.123118    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed7da65-a03d-45d7-aaf0-c79b3920b3ec" containerName="main"
Nov 02 23:11:31.123209 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.123201    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed7da65-a03d-45d7-aaf0-c79b3920b3ec" containerName="main"
Nov 02 23:11:31.126650 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.126633    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.129638 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.129609    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d17d552-d441-4ed5-a5c1-5a4141808868-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.129767 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.129660    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.129836 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.129768    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.129895 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.129840    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.129895 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.129868    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5w94\" (UniqueName: \"kubernetes.io/projected/5d17d552-d441-4ed5-a5c1-5a4141808868-kube-api-access-q5w94\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.130035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.129907    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.130035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.129953    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.130131 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.130049    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.130167 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.130124    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.227289 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.227257    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"]
Nov 02 23:11:31.230856 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.230838    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.230945 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.230865    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5w94\" (UniqueName: \"kubernetes.io/projected/5d17d552-d441-4ed5-a5c1-5a4141808868-kube-api-access-q5w94\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.230945 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.230889    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.230945 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.230908    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.230945 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.230933    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.231169 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.230971    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.231225 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.231212    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d17d552-d441-4ed5-a5c1-5a4141808868-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.231290 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.231244    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.231290 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.231269    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.231290 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.231283    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.231462 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.231317    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.231517 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.231480    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.231670 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.231647    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.231887 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.231868    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d17d552-d441-4ed5-a5c1-5a4141808868-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.233301 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.233280    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.233599 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.233576    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.285502 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.285469    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.285502 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.285494    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5w94\" (UniqueName: \"kubernetes.io/projected/5d17d552-d441-4ed5-a5c1-5a4141808868-kube-api-access-q5w94\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.439589 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.439499    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:31.856467 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.856430    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"]
Nov 02 23:11:31.856890 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:11:31.856852    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d17d552_d441_4ed5_a5c1_5a4141808868.slice/crio-38f6ff74016eba7d9abd151d6965be331f9dfe871ce121b46b9e607e40d069f7 WatchSource:0}: Error finding container 38f6ff74016eba7d9abd151d6965be331f9dfe871ce121b46b9e607e40d069f7: Status 404 returned error can't find the container with id 38f6ff74016eba7d9abd151d6965be331f9dfe871ce121b46b9e607e40d069f7
Nov 02 23:11:31.859640 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.859606    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:11:31.859737 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.859671    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:11:31.859737 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.859703    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:11:31.884871 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:31.884846    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh" event={"ID":"5d17d552-d441-4ed5-a5c1-5a4141808868","Type":"ContainerStarted","Data":"38f6ff74016eba7d9abd151d6965be331f9dfe871ce121b46b9e607e40d069f7"}
Nov 02 23:11:32.892456 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:32.892421    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh" event={"ID":"5d17d552-d441-4ed5-a5c1-5a4141808868","Type":"ContainerStarted","Data":"e89c673aef0fa32412334bcf32446db1b36890b9240345263467d31d72fa3f7b"}
Nov 02 23:11:32.900133 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:32.900105    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:11:33.073856 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:33.073785    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh" podStartSLOduration=2.073763721 podStartE2EDuration="2.073763721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:11:33.051726468 +0000 UTC m=+1132.620081360" watchObservedRunningTime="2026-11-02 23:11:33.073763721 +0000 UTC m=+1132.642118510"
Nov 02 23:11:33.439713 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:33.439679    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:33.441187 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:33.441163    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 23:11:33.441325 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:33.441213    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh" podUID="5d17d552-d441-4ed5-a5c1-5a4141808868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 23:11:34.440239 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:34.440208    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 23:11:34.440643 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:34.440261    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh" podUID="5d17d552-d441-4ed5-a5c1-5a4141808868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 23:11:35.439905 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:35.439811    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 23:11:35.439905 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:35.439869    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh" podUID="5d17d552-d441-4ed5-a5c1-5a4141808868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 23:11:36.440793 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:36.440755    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 23:11:36.441222 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:36.440833    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh" podUID="5d17d552-d441-4ed5-a5c1-5a4141808868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 23:11:37.440797 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:37.440751    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 02 23:11:37.441233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:37.440823    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh" podUID="5d17d552-d441-4ed5-a5c1-5a4141808868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 02 23:11:37.769266 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:37.769194    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:11:38.531765 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:38.531715    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:38.532306 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:38.532244    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:38.534181 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:38.534156    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:11:41.750130 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:11:41.750095    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:11:42.376050 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:42.376003    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:11:43.348244 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.348203    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"]
Nov 02 23:11:43.353755 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.353731    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.402130 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.402086    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.402130 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.402137    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.402398 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.402292    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.402398 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.402329    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.402515 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.402457    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.402515 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.402487    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.402618 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.402525    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51c49a39-5522-4cec-9246-7de8e59d4b9f-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.402670 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.402636    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.402728 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.402715    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d48t\" (UniqueName: \"kubernetes.io/projected/51c49a39-5522-4cec-9246-7de8e59d4b9f-kube-api-access-9d48t\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.448332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.448297    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"]
Nov 02 23:11:43.504109 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.504076    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.504109 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.504118    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.504434 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.504188    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.504434 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.504211    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.504434 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.504239    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51c49a39-5522-4cec-9246-7de8e59d4b9f-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.504434 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.504311    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.504434 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.504375    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d48t\" (UniqueName: \"kubernetes.io/projected/51c49a39-5522-4cec-9246-7de8e59d4b9f-kube-api-access-9d48t\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.504434 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.504414    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.504730 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.504440    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.504730 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.504636    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.504730 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.504683    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.504956 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.504935    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.505070 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.505049    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.505544 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.505524    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51c49a39-5522-4cec-9246-7de8e59d4b9f-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.507027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.506973    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.507125 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.507100    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.561459 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.561432    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d48t\" (UniqueName: \"kubernetes.io/projected/51c49a39-5522-4cec-9246-7de8e59d4b9f-kube-api-access-9d48t\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.561604 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.561436    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:43.666664 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:43.666575    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:44.084500 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:44.084468    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"]
Nov 02 23:11:44.084808 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:11:44.084785    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c49a39_5522_4cec_9246_7de8e59d4b9f.slice/crio-96351999d8992b9eff31cfa37e847961a974372f3cae13f840cadd4219f483ec WatchSource:0}: Error finding container 96351999d8992b9eff31cfa37e847961a974372f3cae13f840cadd4219f483ec: Status 404 returned error can't find the container with id 96351999d8992b9eff31cfa37e847961a974372f3cae13f840cadd4219f483ec
Nov 02 23:11:44.087334 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:44.087305    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:11:44.087414 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:44.087365    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:11:44.087414 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:44.087407    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:11:44.982799 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:44.982765    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff" event={"ID":"51c49a39-5522-4cec-9246-7de8e59d4b9f","Type":"ContainerStarted","Data":"a5a52c2bb72e102fb1330f9a487b8ebdc6688bed91fc90bfff88230064d29b0c"}
Nov 02 23:11:44.983263 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:44.982822    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff" event={"ID":"51c49a39-5522-4cec-9246-7de8e59d4b9f","Type":"ContainerStarted","Data":"96351999d8992b9eff31cfa37e847961a974372f3cae13f840cadd4219f483ec"}
Nov 02 23:11:45.673410 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:45.673372    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:45.679119 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:45.679085    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 23:11:45.679282 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:45.679180    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff" podUID="51c49a39-5522-4cec-9246-7de8e59d4b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 23:11:46.673487 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:46.673443    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 23:11:46.674133 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:46.674098    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff" podUID="51c49a39-5522-4cec-9246-7de8e59d4b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 23:11:46.952857 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:46.952758    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:11:47.672006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:47.668420    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 23:11:47.672006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:47.668500    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff" podUID="51c49a39-5522-4cec-9246-7de8e59d4b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 23:11:48.674003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:48.673399    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 23:11:48.674003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:48.673476    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff" podUID="51c49a39-5522-4cec-9246-7de8e59d4b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 23:11:49.673005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:49.669424    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 23:11:49.673005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:49.669503    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff" podUID="51c49a39-5522-4cec-9246-7de8e59d4b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 23:11:50.667720 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:50.667678    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 02 23:11:50.668210 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:50.667738    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff" podUID="51c49a39-5522-4cec-9246-7de8e59d4b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 02 23:11:51.495789 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:51.495757    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:11:51.672214 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:51.672182    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:51.672639 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:51.672617    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:51.673342 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:51.673324    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:11:51.813731 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:51.813608    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff" podStartSLOduration=8.813594776 podStartE2EDuration="8.813594776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:11:45.148228726 +0000 UTC m=+1144.716583512" watchObservedRunningTime="2026-11-02 23:11:51.813594776 +0000 UTC m=+1151.381949562"
Nov 02 23:11:54.972912 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:54.972875    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"]
Nov 02 23:11:54.976886 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:54.976868    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:55.030992 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.030955    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"]
Nov 02 23:11:55.044522 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.044484    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-59f97a\"/\"distinct-2-w0-59f97a-kserve-self-signed-certs\""
Nov 02 23:11:55.084526 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.084490    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkj85\" (UniqueName: \"kubernetes.io/projected/67d1920b-7d2f-4946-9378-efa0bd270339-kube-api-access-zkj85\") pod \"distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:55.084704 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.084540    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-model-cache\") pod \"distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:55.084745 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.084699    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-tmp-dir\") pod \"distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:55.084835 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.084821    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d1920b-7d2f-4946-9378-efa0bd270339-tls-certs\") pod \"distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:55.084872 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.084844    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-dshm\") pod \"distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:55.084906 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.084870    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-home\") pod \"distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:55.090357 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.090331    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:11:55.186364 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.186323    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-model-cache\") pod \"distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:55.186559 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.186417    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-tmp-dir\") pod \"distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:55.186559 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.186502    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d1920b-7d2f-4946-9378-efa0bd270339-tls-certs\") pod \"distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:55.186559 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.186518    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-dshm\") pod \"distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:55.186559 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.186545    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-home\") pod \"distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:55.186776 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.186594    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkj85\" (UniqueName: \"kubernetes.io/projected/67d1920b-7d2f-4946-9378-efa0bd270339-kube-api-access-zkj85\") pod \"distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:55.186887 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.186864    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-model-cache\") pod \"distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:55.186953 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.186886    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-tmp-dir\") pod \"distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:55.186953 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.186927    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-home\") pod \"distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:55.188889 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.188869    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-dshm\") pod \"distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:55.189160 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.189142    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d1920b-7d2f-4946-9378-efa0bd270339-tls-certs\") pod \"distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:55.241055 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.241008    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkj85\" (UniqueName: \"kubernetes.io/projected/67d1920b-7d2f-4946-9378-efa0bd270339-kube-api-access-zkj85\") pod \"distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") " pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:55.289739 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.289706    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:55.673256 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:55.673227    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"]
Nov 02 23:11:55.673752 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:11:55.673726    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67d1920b_7d2f_4946_9378_efa0bd270339.slice/crio-e0ec62f7ac03ab07e3f84fce96ffa53d240c20c208de76f02cd8a0689a8ed159 WatchSource:0}: Error finding container e0ec62f7ac03ab07e3f84fce96ffa53d240c20c208de76f02cd8a0689a8ed159: Status 404 returned error can't find the container with id e0ec62f7ac03ab07e3f84fce96ffa53d240c20c208de76f02cd8a0689a8ed159
Nov 02 23:11:56.049555 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:56.049513    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" event={"ID":"67d1920b-7d2f-4946-9378-efa0bd270339","Type":"ContainerStarted","Data":"f94a39b8d0087ee97f2c07169ce6030c74fa0b8debfb0e0e08bb54ff3069c164"}
Nov 02 23:11:56.049957 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:56.049568    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" event={"ID":"67d1920b-7d2f-4946-9378-efa0bd270339","Type":"ContainerStarted","Data":"e0ec62f7ac03ab07e3f84fce96ffa53d240c20c208de76f02cd8a0689a8ed159"}
Nov 02 23:11:56.160794 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:56.160738    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" podStartSLOduration=2.160724765 podStartE2EDuration="2.160724765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:11:56.158518685 +0000 UTC m=+1155.726873473" watchObservedRunningTime="2026-11-02 23:11:56.160724765 +0000 UTC m=+1155.729079552"
Nov 02 23:11:56.204327 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:56.204297    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:11:56.290614 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:56.290578    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:11:57.202895 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.202860    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"]
Nov 02 23:11:57.207058 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.207041    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.300125 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.300093    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"]
Nov 02 23:11:57.308615 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.308575    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.308783 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.308693    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.308783 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.308741    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.308923 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.308784    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.308923 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.308860    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.309064 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.308920    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.309064 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.308968    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sljjp\" (UniqueName: \"kubernetes.io/projected/33529202-6346-4b2a-b3ad-59dc8b8fa59a-kube-api-access-sljjp\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.309064 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.309016    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.309222 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.309070    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.410144 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.410102    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.410144 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.410153    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.410420 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.410181    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.410420 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.410229    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.410420 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.410275    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.410420 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.410323    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljjp\" (UniqueName: \"kubernetes.io/projected/33529202-6346-4b2a-b3ad-59dc8b8fa59a-kube-api-access-sljjp\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.410420 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.410356    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.410680 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.410423    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.410680 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.410460    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.410760 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.410730    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.410795 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.410761    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.410925 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.410900    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.411119 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.411100    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.411492 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.411466    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.412618 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.412597    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.412991 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.412954    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.465192 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.465114    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.465480 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.465450    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljjp\" (UniqueName: \"kubernetes.io/projected/33529202-6346-4b2a-b3ad-59dc8b8fa59a-kube-api-access-sljjp\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.521223 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.521190    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:57.937662 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.937636    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"]
Nov 02 23:11:57.939306 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:11:57.939283    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33529202_6346_4b2a_b3ad_59dc8b8fa59a.slice/crio-efbc2b015a9535bfb8cbed9352d5a0f99b1fc0de6b4ce0b6d79f6abce833c8b1 WatchSource:0}: Error finding container efbc2b015a9535bfb8cbed9352d5a0f99b1fc0de6b4ce0b6d79f6abce833c8b1: Status 404 returned error can't find the container with id efbc2b015a9535bfb8cbed9352d5a0f99b1fc0de6b4ce0b6d79f6abce833c8b1
Nov 02 23:11:57.942071 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.942037    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:11:57.942145 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.942109    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:11:57.942183 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:57.942140    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:11:58.063249 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:58.063212    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5" event={"ID":"33529202-6346-4b2a-b3ad-59dc8b8fa59a","Type":"ContainerStarted","Data":"8defac415c8184cb5b308d823b9deee784c3e10551f74937907bf56636458473"}
Nov 02 23:11:58.063436 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:58.063278    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5" event={"ID":"33529202-6346-4b2a-b3ad-59dc8b8fa59a","Type":"ContainerStarted","Data":"efbc2b015a9535bfb8cbed9352d5a0f99b1fc0de6b4ce0b6d79f6abce833c8b1"}
Nov 02 23:11:58.175609 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:58.175543    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5" podStartSLOduration=1.175521903 podStartE2EDuration="1.175521903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:11:58.17433186 +0000 UTC m=+1157.742686649" watchObservedRunningTime="2026-11-02 23:11:58.175521903 +0000 UTC m=+1157.743876691"
Nov 02 23:11:58.521380 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:58.521344    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:11:59.523714 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:59.523675    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 23:11:59.524241 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:59.523744    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5" podUID="33529202-6346-4b2a-b3ad-59dc8b8fa59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 23:11:59.627219 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:11:59.627188    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:12:00.522150 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:00.522113    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 23:12:00.522364 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:00.522171    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5" podUID="33529202-6346-4b2a-b3ad-59dc8b8fa59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 23:12:00.796569 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:00.796483    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:12:01.521788 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:01.521756    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 23:12:01.521953 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:01.521807    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5" podUID="33529202-6346-4b2a-b3ad-59dc8b8fa59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 23:12:02.522088 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:02.522054    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 23:12:02.522578 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:02.522117    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5" podUID="33529202-6346-4b2a-b3ad-59dc8b8fa59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 23:12:03.521942 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:03.521907    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 23:12:03.522140 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:03.521993    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5" podUID="33529202-6346-4b2a-b3ad-59dc8b8fa59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 23:12:04.523004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:04.522648    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 02 23:12:04.523004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:04.522719    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5" podUID="33529202-6346-4b2a-b3ad-59dc8b8fa59a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 02 23:12:05.308113 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:05.308068    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:12:05.588509 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:05.588419    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:12:05.653056 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:05.652728    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:12:06.181137 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:06.181106    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:12:06.233265 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:06.233236    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:12:06.233650 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:06.233628    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:12:06.235278 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:06.235259    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:12:10.167551 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:10.167513    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:12:11.750732 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:12:11.750695    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:12:12.346668 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.346621    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"]
Nov 02 23:12:12.352971 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.352944    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.436399 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.436363    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.436569 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.436413    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvt46\" (UniqueName: \"kubernetes.io/projected/09f13590-7f6c-4c1c-90c2-b59751b42720-kube-api-access-rvt46\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.436640 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.436602    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.436707 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.436692    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.436780 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.436767    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.436883 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.436865    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f13590-7f6c-4c1c-90c2-b59751b42720-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.436973 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.436957    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.437062 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.437048    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.437112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.437078    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.538330 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.538291    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvt46\" (UniqueName: \"kubernetes.io/projected/09f13590-7f6c-4c1c-90c2-b59751b42720-kube-api-access-rvt46\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.539613 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.539560    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.539763 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.539636    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.539763 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.539743    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.539893 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.539793    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f13590-7f6c-4c1c-90c2-b59751b42720-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.539893 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.539861    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.540010 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.539924    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.540010 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.539954    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.540118 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.540029    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.540933 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.540901    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f13590-7f6c-4c1c-90c2-b59751b42720-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.541227 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.541206    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.541455 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.541437    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.541675 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.541657    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.542604 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.542578    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.545333 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.545289    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.546182 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.545655    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.552658 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.552619    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"]
Nov 02 23:12:12.593416 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.593384    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvt46\" (UniqueName: \"kubernetes.io/projected/09f13590-7f6c-4c1c-90c2-b59751b42720-kube-api-access-rvt46\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.597697 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.597629    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:12.666773 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:12.666735    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:13.082907 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:13.082881    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"]
Nov 02 23:12:13.083258 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:12:13.082884    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f13590_7f6c_4c1c_90c2_b59751b42720.slice/crio-220b27fab968abab7505fb48e69af0bd7254d8702ff0c74b8b3413c722f004e3 WatchSource:0}: Error finding container 220b27fab968abab7505fb48e69af0bd7254d8702ff0c74b8b3413c722f004e3: Status 404 returned error can't find the container with id 220b27fab968abab7505fb48e69af0bd7254d8702ff0c74b8b3413c722f004e3
Nov 02 23:12:13.085360 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:13.085330    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:12:13.085442 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:13.085391    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:12:13.085442 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:13.085421    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:12:13.217058 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:13.217020    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96" event={"ID":"09f13590-7f6c-4c1c-90c2-b59751b42720","Type":"ContainerStarted","Data":"2d35dec0330a9e301a327b187b2f15a71161acc379b0df672ee1c591d40b3b2c"}
Nov 02 23:12:13.217250 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:13.217087    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96" event={"ID":"09f13590-7f6c-4c1c-90c2-b59751b42720","Type":"ContainerStarted","Data":"220b27fab968abab7505fb48e69af0bd7254d8702ff0c74b8b3413c722f004e3"}
Nov 02 23:12:13.334703 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:13.334592    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96" podStartSLOduration=1.334575843 podStartE2EDuration="1.334575843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:13.331899428 +0000 UTC m=+1172.900254214" watchObservedRunningTime="2026-11-02 23:12:13.334575843 +0000 UTC m=+1172.902930630"
Nov 02 23:12:13.667204 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:13.667110    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:14.678144 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:14.678101    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 23:12:14.678666 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:14.678176    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96" podUID="09f13590-7f6c-4c1c-90c2-b59751b42720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 23:12:14.779193 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:14.776132    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:12:15.678006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:15.677395    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 23:12:15.678006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:15.677458    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96" podUID="09f13590-7f6c-4c1c-90c2-b59751b42720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 23:12:16.672427 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:16.672387    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 23:12:16.672717 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:16.672688    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96" podUID="09f13590-7f6c-4c1c-90c2-b59751b42720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 23:12:17.673005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:17.670347    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 23:12:17.673005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:17.670409    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96" podUID="09f13590-7f6c-4c1c-90c2-b59751b42720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 23:12:18.674411 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:18.674371    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 23:12:18.675062 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:18.675027    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96" podUID="09f13590-7f6c-4c1c-90c2-b59751b42720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 23:12:19.565552 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:19.565515    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:12:19.667269 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:19.667223    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 02 23:12:19.667507 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:19.667302    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96" podUID="09f13590-7f6c-4c1c-90c2-b59751b42720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 02 23:12:20.672129 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:20.672096    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:20.672532 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:20.672386    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:20.673042 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:20.673027    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:12:24.171529 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:24.171496    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:12:26.869306 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:26.869266    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"]
Nov 02 23:12:26.889002 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:26.887344    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.043013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.022001    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.043013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.022139    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50895b59-7759-4a46-aa8f-a1d691c255de-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.043013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.022191    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.043013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.022218    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50895b59-7759-4a46-aa8f-a1d691c255de-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.043013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.022258    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5vhs\" (UniqueName: \"kubernetes.io/projected/50895b59-7759-4a46-aa8f-a1d691c255de-kube-api-access-d5vhs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.043013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.022290    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50895b59-7759-4a46-aa8f-a1d691c255de-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.043013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.022329    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.043013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.022390    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.043013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.022421    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.186058 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.169363    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.186462 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.186426    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50895b59-7759-4a46-aa8f-a1d691c255de-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.195150 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.195112    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.195416 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.195358    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50895b59-7759-4a46-aa8f-a1d691c255de-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.220125 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.220084    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5vhs\" (UniqueName: \"kubernetes.io/projected/50895b59-7759-4a46-aa8f-a1d691c255de-kube-api-access-d5vhs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.220401 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.220375    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50895b59-7759-4a46-aa8f-a1d691c255de-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.220553 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.220537    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.220714 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.220698    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.220830 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.220814    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.362027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.351745    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:27.391027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.363040    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.391027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.363760    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.391027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.364460    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50895b59-7759-4a46-aa8f-a1d691c255de-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.391027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.364959    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.391027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.365238    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.391027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.369079    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" podUID="845035ca-c55e-4e32-a19a-dc3617e3d638" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 23:12:27.391027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.374233    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.655001 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.651739    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf" podUID="2f110c17-1134-4f5c-b9a6-51abb2ac481f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:27.655001 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.652781    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:27.655001 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.654667    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"]
Nov 02 23:12:27.666245 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.666209    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:12:27.742541 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.687795    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9" podUID="43e3d0d6-1a8e-4536-bc46-b5785124405a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:27.851099 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.743641    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh" podUID="e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:27.858070 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.769230    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5vhs\" (UniqueName: \"kubernetes.io/projected/50895b59-7759-4a46-aa8f-a1d691c255de-kube-api-access-d5vhs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.858070 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.788000    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50895b59-7759-4a46-aa8f-a1d691c255de-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.858070 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.806111    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb" podUID="aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:27.934056 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.903564    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j" podUID="f483c007-731c-42e7-a7f2-9b7877fbba41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:27.934919 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.913621    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50895b59-7759-4a46-aa8f-a1d691c255de-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:27.996779 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:27.996721    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:12:28.117611 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.117567    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:28.672230 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.662964    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"]
Nov 02 23:12:28.672230 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.669374    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:12:28.672230 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.669459    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:12:28.672230 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.669500    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:12:28.740830 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.740807    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"]
Nov 02 23:12:28.746365 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.746344    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.821228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.819153    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.821228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.819327    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.821228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.819405    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.821228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.819448    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkprk\" (UniqueName: \"kubernetes.io/projected/8a3538a4-c25e-4d3d-97e5-267a770bbb43-kube-api-access-zkprk\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.821228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.819699    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.821228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.819782    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.821228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.819810    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.821228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.819880    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.821228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.819991    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.836589 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.836453    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"]
Nov 02 23:12:28.844241 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.842239    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w" event={"ID":"50895b59-7759-4a46-aa8f-a1d691c255de","Type":"ContainerStarted","Data":"e0b8cb8ebad878d355e5d3cdcd1bc4cc6acd8333220d5807dab36e2e89790bed"}
Nov 02 23:12:28.844241 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.842298    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w" event={"ID":"50895b59-7759-4a46-aa8f-a1d691c255de","Type":"ContainerStarted","Data":"d625dc49df74a9186dbe1c67fbda7081183b76bface47f1f9549b047f2d5a435"}
Nov 02 23:12:28.921396 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.921314    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.921551 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.921416    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.921551 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.921453    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.921551 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.921469    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkprk\" (UniqueName: \"kubernetes.io/projected/8a3538a4-c25e-4d3d-97e5-267a770bbb43-kube-api-access-zkprk\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.921676 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.921555    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.921676 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.921631    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.921676 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.921651    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.921836 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.921741    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.921836 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.921788    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.921924 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.921840    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-workload-certs\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.922310 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.922130    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istiod-ca-cert\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.922310 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.922191    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-credential-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.922310 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.922196    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-workload-socket\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.922536 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.922345    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-data\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.924215 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.924192    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-envoy\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.924417 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.924400    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-podinfo\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.973685 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.973621    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w" podStartSLOduration=2.97360294 podStartE2EDuration="2.97360294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:28.970796669 +0000 UTC m=+1188.539151458" watchObservedRunningTime="2026-11-02 23:12:28.97360294 +0000 UTC m=+1188.541957730"
Nov 02 23:12:28.977400 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.977371    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-token\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:28.978038 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:28.978014    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkprk\" (UniqueName: \"kubernetes.io/projected/8a3538a4-c25e-4d3d-97e5-267a770bbb43-kube-api-access-zkprk\") pod \"e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") " pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:29.061147 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:29.061116    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:29.119612 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:29.119128    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:29.121151 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:29.121128    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 23:12:29.121227 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:29.121181    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w" podUID="50895b59-7759-4a46-aa8f-a1d691c255de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 23:12:29.484794 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:29.484748    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"]
Nov 02 23:12:29.486518 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:12:29.486492    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3538a4_c25e_4d3d_97e5_267a770bbb43.slice/crio-8705820a7bd01ee1d31ba667d615c40cc2e7e2dda59972785d9deebd0ae78d30 WatchSource:0}: Error finding container 8705820a7bd01ee1d31ba667d615c40cc2e7e2dda59972785d9deebd0ae78d30: Status 404 returned error can't find the container with id 8705820a7bd01ee1d31ba667d615c40cc2e7e2dda59972785d9deebd0ae78d30
Nov 02 23:12:29.490721 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:29.490679    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:12:29.490904 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:29.490774    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:12:29.490904 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:29.490816    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:12:29.493346 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:29.493319    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:12:29.852056 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:29.852017    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg" event={"ID":"8a3538a4-c25e-4d3d-97e5-267a770bbb43","Type":"ContainerStarted","Data":"402ec30c6b2c75c44c115e4b7a438712813a599b5ae1a25989bc373a7e80b004"}
Nov 02 23:12:29.852233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:29.852084    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg" event={"ID":"8a3538a4-c25e-4d3d-97e5-267a770bbb43","Type":"ContainerStarted","Data":"8705820a7bd01ee1d31ba667d615c40cc2e7e2dda59972785d9deebd0ae78d30"}
Nov 02 23:12:29.987513 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:29.987463    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg" podStartSLOduration=1.987447355 podStartE2EDuration="1.987447355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:29.973497175 +0000 UTC m=+1189.541851965" watchObservedRunningTime="2026-11-02 23:12:29.987447355 +0000 UTC m=+1189.555802142"
Nov 02 23:12:30.077115 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:30.075267    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:30.082485 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:30.082446    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 23:12:30.082692 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:30.082509    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg" podUID="8a3538a4-c25e-4d3d-97e5-267a770bbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 23:12:30.119215 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:30.119129    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 23:12:30.119215 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:30.119179    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w" podUID="50895b59-7759-4a46-aa8f-a1d691c255de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 23:12:31.076004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:31.075448    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 23:12:31.076004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:31.075518    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg" podUID="8a3538a4-c25e-4d3d-97e5-267a770bbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 23:12:31.118855 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:31.118819    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 23:12:31.119042 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:31.118872    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w" podUID="50895b59-7759-4a46-aa8f-a1d691c255de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 23:12:32.075009 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:32.074464    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 23:12:32.075009 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:32.074532    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg" podUID="8a3538a4-c25e-4d3d-97e5-267a770bbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 23:12:32.119190 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:32.119147    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 23:12:32.119603 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:32.119212    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w" podUID="50895b59-7759-4a46-aa8f-a1d691c255de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 23:12:33.081004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:33.073495    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 23:12:33.081004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:33.073559    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg" podUID="8a3538a4-c25e-4d3d-97e5-267a770bbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 23:12:33.118833 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:33.118795    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 23:12:33.119038 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:33.118858    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w" podUID="50895b59-7759-4a46-aa8f-a1d691c255de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 23:12:34.066440 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:34.066398    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 23:12:34.079162 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:34.079101    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg" podUID="8a3538a4-c25e-4d3d-97e5-267a770bbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 23:12:34.118727 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:34.118689    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 23:12:34.118951 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:34.118931    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w" podUID="50895b59-7759-4a46-aa8f-a1d691c255de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 23:12:35.076005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:35.067403    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 23:12:35.076005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:35.067467    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg" podUID="8a3538a4-c25e-4d3d-97e5-267a770bbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 23:12:35.119023 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:35.118961    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 02 23:12:35.119198 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:35.119048    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w" podUID="50895b59-7759-4a46-aa8f-a1d691c255de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 02 23:12:35.147873 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:35.147817    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:12:36.061938 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:36.061899    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 02 23:12:36.062138 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:36.061966    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg" podUID="8a3538a4-c25e-4d3d-97e5-267a770bbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 02 23:12:36.485520 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:36.485419    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:36.486065 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:36.486040    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:36.487129 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:36.487105    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:12:37.304906 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:37.304872    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:37.305339 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:37.305316    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:37.306113 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:37.306093    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:12:39.927510 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:39.927475    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:12:41.750686 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:12:41.750652    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:12:41.978876 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:41.978846    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"]
Nov 02 23:12:41.984078 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:41.984055    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.082469 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.082377    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.082469 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.082431    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.082720 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.082540    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.082720 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.082602    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.082720 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.082663    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.082887 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.082741    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.082887 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.082777    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlh45\" (UniqueName: \"kubernetes.io/projected/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-kube-api-access-wlh45\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.082887 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.082835    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.083092 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.082939    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.087310 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.087283    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"]
Nov 02 23:12:42.185664 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.185619    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.185870 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.185740    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.185870 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.185782    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.185870 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.185812    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.185870 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.185860    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.186091 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.185923    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.186091 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.186027    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.186091 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.186063    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlh45\" (UniqueName: \"kubernetes.io/projected/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-kube-api-access-wlh45\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.186239 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.186126    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.186686 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.186658    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-workload-certs\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.186943 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.186922    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-workload-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.189033 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.188105    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-data\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.189033 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.188714    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istiod-ca-cert\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.189315 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.189280    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-credential-socket\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.190482 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.190459    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-podinfo\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.191230 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.191211    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-envoy\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.243998 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.243960    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-token\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.244184 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.244161    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlh45\" (UniqueName: \"kubernetes.io/projected/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-kube-api-access-wlh45\") pod \"e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") " pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.298053 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.298020    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:42.713570 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.713483    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"]
Nov 02 23:12:42.748670 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:12:42.748643    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b6e169_e4d8_4a5d_b5a2_88d4c424503e.slice/crio-e8b82da3f26b258cc5af6b9dca5abf69376d8fd5bc427c5fb12492ed0b505a64 WatchSource:0}: Error finding container e8b82da3f26b258cc5af6b9dca5abf69376d8fd5bc427c5fb12492ed0b505a64: Status 404 returned error can't find the container with id e8b82da3f26b258cc5af6b9dca5abf69376d8fd5bc427c5fb12492ed0b505a64
Nov 02 23:12:42.751192 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.751159    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:12:42.751601 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.751241    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:12:42.751601 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.751288    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:12:42.988970 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.988886    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws" event={"ID":"72b6e169-e4d8-4a5d-b5a2-88d4c424503e","Type":"ContainerStarted","Data":"15c419604dc515d6a1bc875013b3abd7fdfd89642fc82d30be32aa48ca3aaa22"}
Nov 02 23:12:42.988970 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:42.988941    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws" event={"ID":"72b6e169-e4d8-4a5d-b5a2-88d4c424503e","Type":"ContainerStarted","Data":"e8b82da3f26b258cc5af6b9dca5abf69376d8fd5bc427c5fb12492ed0b505a64"}
Nov 02 23:12:43.104528 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:43.104466    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws" podStartSLOduration=2.104449076 podStartE2EDuration="2.104449076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:12:43.100674621 +0000 UTC m=+1202.669029409" watchObservedRunningTime="2026-11-02 23:12:43.104449076 +0000 UTC m=+1202.672803864"
Nov 02 23:12:43.298494 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:43.298450    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:43.300454 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:43.300429    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 23:12:43.300581 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:43.300483    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws" podUID="72b6e169-e4d8-4a5d-b5a2-88d4c424503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 23:12:44.298482 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:44.298447    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 23:12:44.298890 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:44.298500    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws" podUID="72b6e169-e4d8-4a5d-b5a2-88d4c424503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 23:12:44.552559 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:44.552452    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:12:45.298510 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:45.298475    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 23:12:45.298911 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:45.298536    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws" podUID="72b6e169-e4d8-4a5d-b5a2-88d4c424503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 23:12:46.298619 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:46.298580    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 23:12:46.299073 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:46.298636    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws" podUID="72b6e169-e4d8-4a5d-b5a2-88d4c424503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 23:12:47.298610 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:47.298569    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 23:12:47.298799 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:47.298629    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws" podUID="72b6e169-e4d8-4a5d-b5a2-88d4c424503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 23:12:48.299266 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:48.299232    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 23:12:48.299670 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:48.299287    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws" podUID="72b6e169-e4d8-4a5d-b5a2-88d4c424503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 23:12:49.140731 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:49.140692    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:12:49.299206 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:49.299167    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 02 23:12:49.299389 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:49.299237    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws" podUID="72b6e169-e4d8-4a5d-b5a2-88d4c424503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 02 23:12:50.302778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:50.302748    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:50.303410 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:50.303165    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:50.303911 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:50.303891    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:12:53.063158 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:53.063097    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" podUID="845035ca-c55e-4e32-a19a-dc3617e3d638" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:53.176216 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:53.175100    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 02 23:12:53.356112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:53.354256    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 23:12:53.356112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:53.354326    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg" podUID="8a3538a4-c25e-4d3d-97e5-267a770bbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:53.511206 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:53.511138    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:53.584144 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:53.584085    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf" podUID="2f110c17-1134-4f5c-b9a6-51abb2ac481f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:53.679229 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:53.679074    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9" podUID="43e3d0d6-1a8e-4536-bc46-b5785124405a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 23:12:53.736501 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:53.736451    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh" podUID="e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:53.811243 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:53.808077    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb" podUID="aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:53.903005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:53.900776    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j" podUID="f483c007-731c-42e7-a7f2-9b7877fbba41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:54.079006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:54.063881    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" podUID="845035ca-c55e-4e32-a19a-dc3617e3d638" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:54.079006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:54.064407    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:12:54.196006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:54.178100    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:54.196006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:54.179163    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:12:54.439161 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:54.389182    4294 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb" podUID="aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:54.546006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:54.529433    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:54.546006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:54.529948    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:12:54.632847 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:54.597139    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf" podUID="2f110c17-1134-4f5c-b9a6-51abb2ac481f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:54.662188 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:54.662133    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 23:12:54.662540 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:54.662505    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh" podUID="5d17d552-d441-4ed5-a5c1-5a4141808868" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:54.689483 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:54.689388    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9" podUID="43e3d0d6-1a8e-4536-bc46-b5785124405a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:54.747306 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:54.747213    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh" podUID="e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:54.857875 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:54.832345    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb" podUID="aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:54.959103 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:54.958970    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j" podUID="f483c007-731c-42e7-a7f2-9b7877fbba41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 02 23:12:55.047155 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:55.047108    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 23:12:55.047501 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:55.047472    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq" podUID="52a13074-c991-4d08-9bfd-c132ffab4c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:55.102005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:55.081269    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" podUID="845035ca-c55e-4e32-a19a-dc3617e3d638" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:55.245062 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:55.220506    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:12:55.651062 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:55.598272    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:12:55.651062 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:55.598325    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:12:55.651062 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:55.598364    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:12:55.651062 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:55.598396    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:12:55.651062 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:55.279051    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:55.651062 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:55.598060    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf" podUID="2f110c17-1134-4f5c-b9a6-51abb2ac481f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:55.651062 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:55.543093    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:55.756017 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:55.750456    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9" podUID="43e3d0d6-1a8e-4536-bc46-b5785124405a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 23:12:55.823009 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:55.773758    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh" podUID="e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 02 23:12:55.965064 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:55.871014    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb" podUID="aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:56.002387 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:55.966342    4294 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:56.002658 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:55.967034    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j" podUID="f483c007-731c-42e7-a7f2-9b7877fbba41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:56.136030 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:56.094035    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" podUID="845035ca-c55e-4e32-a19a-dc3617e3d638" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 23:12:56.334642 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:56.334605    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:12:56.369690 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:56.353175    4294 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded"
Nov 02 23:12:56.557696 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:56.557664    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:12:56.607061 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:56.602186    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 02 23:12:56.654355 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:56.641044    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf" podUID="2f110c17-1134-4f5c-b9a6-51abb2ac481f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:56.654704 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:56.654607    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:12:56.665763 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:56.665734    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:12:56.668137 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:56.668114    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:12:56.673706 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:56.673686    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:12:56.679066 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:56.678461    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:12:56.682827 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:56.682810    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:12:56.686382 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:56.686363    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:12:56.699326 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:56.699297    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:12:57.739653 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:12:57.739621    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:00.138422 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:00.138386    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:00.309878 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:00.309847    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:00.974204 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:00.974169    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:01.358186 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:01.358153    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:01.565530 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:01.565498    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:05.652803 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:05.652758    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:10.043761 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.043722    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:10.273618 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.273583    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:10.539572 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.539538    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"]
Nov 02 23:13:10.543910 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.543885    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.633288 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.633240    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.638848 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.638810    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.639290 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.639241    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.639610 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.639582    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6735e93-95bc-4514-8739-31725bf4fd5d-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.639817 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.639800    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.640002 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.639962    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.640230 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.640160    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.640489 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.640460    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.640752 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.640723    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2kzk\" (UniqueName: \"kubernetes.io/projected/d6735e93-95bc-4514-8739-31725bf4fd5d-kube-api-access-s2kzk\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.676030 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.671613    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e953257b-openshift-default-dockercfg-9qzkk\""
Nov 02 23:13:10.699317 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.677375    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"]
Nov 02 23:13:10.699317 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.678277    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:10.752476 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.750925    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.752476 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.751104    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6735e93-95bc-4514-8739-31725bf4fd5d-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.752476 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.751176    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.752476 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.751244    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.752476 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.751319    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.752476 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.751417    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.752476 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.751427    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.752476 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.751533    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2kzk\" (UniqueName: \"kubernetes.io/projected/d6735e93-95bc-4514-8739-31725bf4fd5d-kube-api-access-s2kzk\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.752476 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.751579    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.752476 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.751715    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.752476 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.752328    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6735e93-95bc-4514-8739-31725bf4fd5d-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.754245 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.752585    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.754245 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.752647    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.754245 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.752855    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.755273 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.755241    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.755616 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.755594    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.805910 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.805824    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.806322 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.806285    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2kzk\" (UniqueName: \"kubernetes.io/projected/d6735e93-95bc-4514-8739-31725bf4fd5d-kube-api-access-s2kzk\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:10.860937 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:10.860897    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:11.175664 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:11.175582    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:11.331559 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:11.331524    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"]
Nov 02 23:13:11.332058 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:13:11.332032    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6735e93_95bc_4514_8739_31725bf4fd5d.slice/crio-088ecc0d04dd622af85f81521df68588d22ac8a8ef1fafa2ec6d439f738a519d WatchSource:0}: Error finding container 088ecc0d04dd622af85f81521df68588d22ac8a8ef1fafa2ec6d439f738a519d: Status 404 returned error can't find the container with id 088ecc0d04dd622af85f81521df68588d22ac8a8ef1fafa2ec6d439f738a519d
Nov 02 23:13:11.334617 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:11.334584    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:13:11.334724 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:11.334658    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:13:11.334724 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:11.334703    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:13:11.755616 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:13:11.755579    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:13:11.782347 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:11.782309    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn" event={"ID":"d6735e93-95bc-4514-8739-31725bf4fd5d","Type":"ContainerStarted","Data":"94bddf21811a60a483efc68980b67ed91f0d4b84103c92251a212498aad9ba00"}
Nov 02 23:13:11.782566 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:11.782368    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn" event={"ID":"d6735e93-95bc-4514-8739-31725bf4fd5d","Type":"ContainerStarted","Data":"088ecc0d04dd622af85f81521df68588d22ac8a8ef1fafa2ec6d439f738a519d"}
Nov 02 23:13:11.864752 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:11.864708    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:11.896277 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:11.896221    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn" podStartSLOduration=1.896205842 podStartE2EDuration="1.896205842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:11.894056158 +0000 UTC m=+1231.462410947" watchObservedRunningTime="2026-11-02 23:13:11.896205842 +0000 UTC m=+1231.464560698"
Nov 02 23:13:12.867277 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:12.867245    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:13.816233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:13.816200    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:13.817513 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:13.817476    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:13:14.894274 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:14.894241    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:15.101811 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:15.101781    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:15.303616 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:15.303570    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:16.955827 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:16.955791    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"]
Nov 02 23:13:16.961634 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:16.961611    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:16.981122 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:16.981091    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67334c5b-8d17-478e-8248-0762c0e1653c-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:16.981252 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:16.981131    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:16.981252 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:16.981235    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:16.981323 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:16.981275    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:16.981377 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:16.981362    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67334c5b-8d17-478e-8248-0762c0e1653c-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:16.981469 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:16.981456    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnvzn\" (UniqueName: \"kubernetes.io/projected/67334c5b-8d17-478e-8248-0762c0e1653c-kube-api-access-nnvzn\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:16.981512 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:16.981479    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67334c5b-8d17-478e-8248-0762c0e1653c-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:16.981572 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:16.981561    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:16.981610 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:16.981590    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:17.062912 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.062874    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"]
Nov 02 23:13:17.082850 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.082813    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:17.083060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.082880    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:17.083060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.082929    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67334c5b-8d17-478e-8248-0762c0e1653c-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:17.083060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.082961    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:17.083060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.083039    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:17.083280 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.083079    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:17.083280 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.083142    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67334c5b-8d17-478e-8248-0762c0e1653c-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:17.083280 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.083254    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnvzn\" (UniqueName: \"kubernetes.io/projected/67334c5b-8d17-478e-8248-0762c0e1653c-kube-api-access-nnvzn\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:17.083280 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.083256    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:17.083493 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.083288    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67334c5b-8d17-478e-8248-0762c0e1653c-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:17.083493 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.083295    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:17.083615 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.083518    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:17.083994 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.083959    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67334c5b-8d17-478e-8248-0762c0e1653c-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:17.084328 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.084288    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:17.086664 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.086621    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:17.086664 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.086641    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67334c5b-8d17-478e-8248-0762c0e1653c-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:17.139361 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.139322    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67334c5b-8d17-478e-8248-0762c0e1653c-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:17.139682 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.139663    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnvzn\" (UniqueName: \"kubernetes.io/projected/67334c5b-8d17-478e-8248-0762c0e1653c-kube-api-access-nnvzn\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:17.275590 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.275559    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:17.691317 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.691289    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"]
Nov 02 23:13:17.691458 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:13:17.691284    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67334c5b_8d17_478e_8248_0762c0e1653c.slice/crio-c3b9cf0780080e552e9cf300ada619af2aab55f9ad7a1a05872f03e8774aca76 WatchSource:0}: Error finding container c3b9cf0780080e552e9cf300ada619af2aab55f9ad7a1a05872f03e8774aca76: Status 404 returned error can't find the container with id c3b9cf0780080e552e9cf300ada619af2aab55f9ad7a1a05872f03e8774aca76
Nov 02 23:13:17.693717 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.693687    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:13:17.693804 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.693751    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:13:17.693804 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.693797    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:13:17.871225 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.871180    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp" event={"ID":"67334c5b-8d17-478e-8248-0762c0e1653c","Type":"ContainerStarted","Data":"0824e99f6eb8e9ee9e5ae119af94a660358274fb1d6f8228ed5d4ea2707d2556"}
Nov 02 23:13:17.871430 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.871249    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp" event={"ID":"67334c5b-8d17-478e-8248-0762c0e1653c","Type":"ContainerStarted","Data":"c3b9cf0780080e552e9cf300ada619af2aab55f9ad7a1a05872f03e8774aca76"}
Nov 02 23:13:17.988809 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:17.988698    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp" podStartSLOduration=1.988681178 podStartE2EDuration="1.988681178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:17.986048629 +0000 UTC m=+1237.554403416" watchObservedRunningTime="2026-11-02 23:13:17.988681178 +0000 UTC m=+1237.557036026"
Nov 02 23:13:18.047788 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:18.047756    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:18.213027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:18.212969    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:18.276301 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:18.276264    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:18.277950 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:18.277923    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 23:13:18.278102 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:18.277970    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp" podUID="67334c5b-8d17-478e-8248-0762c0e1653c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 23:13:18.634058 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:18.628121    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8"]
Nov 02 23:13:18.641795 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:18.641067    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8"
Nov 02 23:13:18.642106 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:18.640940    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8"]
Nov 02 23:13:18.709041 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:18.705428    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e953257b-dockercfg-bmwt4\""
Nov 02 23:13:18.709041 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:18.708498    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23f0ba4-8743-4fae-bdbe-307266213d9e-plugins-config-volume\") pod \"payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8\" (UID: \"c23f0ba4-8743-4fae-bdbe-307266213d9e\") " pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8"
Nov 02 23:13:18.709041 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:18.708542    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-662dv\" (UniqueName: \"kubernetes.io/projected/c23f0ba4-8743-4fae-bdbe-307266213d9e-kube-api-access-662dv\") pod \"payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8\" (UID: \"c23f0ba4-8743-4fae-bdbe-307266213d9e\") " pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8"
Nov 02 23:13:18.753111 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:18.744723    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"]
Nov 02 23:13:18.753111 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:18.744951    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl" podUID="96fc6247-0b37-40df-b795-2c799c981876" containerName="istio-proxy" containerID="cri-o://c0d05befac874f4db919144cda4523025522f5347d227f20168f0a4ff352e1dd" gracePeriod=30
Nov 02 23:13:18.774410 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:18.708847    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e953257b\""
Nov 02 23:13:18.823005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:18.822326    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23f0ba4-8743-4fae-bdbe-307266213d9e-plugins-config-volume\") pod \"payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8\" (UID: \"c23f0ba4-8743-4fae-bdbe-307266213d9e\") " pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8"
Nov 02 23:13:18.823005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:18.822373    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-662dv\" (UniqueName: \"kubernetes.io/projected/c23f0ba4-8743-4fae-bdbe-307266213d9e-kube-api-access-662dv\") pod \"payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8\" (UID: \"c23f0ba4-8743-4fae-bdbe-307266213d9e\") " pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8"
Nov 02 23:13:18.824705 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:18.824559    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23f0ba4-8743-4fae-bdbe-307266213d9e-plugins-config-volume\") pod \"payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8\" (UID: \"c23f0ba4-8743-4fae-bdbe-307266213d9e\") " pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8"
Nov 02 23:13:18.925071 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:18.922737    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-662dv\" (UniqueName: \"kubernetes.io/projected/c23f0ba4-8743-4fae-bdbe-307266213d9e-kube-api-access-662dv\") pod \"payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8\" (UID: \"c23f0ba4-8743-4fae-bdbe-307266213d9e\") " pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8"
Nov 02 23:13:19.050047 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:19.046240    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8"
Nov 02 23:13:19.137001 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:19.130861    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:19.280002 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:19.279421    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 23:13:19.280002 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:19.279485    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp" podUID="67334c5b-8d17-478e-8248-0762c0e1653c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 23:13:20.259953 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:20.259918    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8"]
Nov 02 23:13:20.296011 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:20.295941    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 23:13:20.296307 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:20.296274    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp" podUID="67334c5b-8d17-478e-8248-0762c0e1653c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 23:13:20.488111 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:20.484677    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:20.579006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:20.576126    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb" podUID="aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:20.622197 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:20.622160    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 23:13:20.622506 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:20.622457    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg" podUID="b007881f-8d1a-487c-ab6f-0ad740dd047a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:20.768474 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:20.766522    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j" podUID="f483c007-731c-42e7-a7f2-9b7877fbba41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:20.769826 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:20.768840    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf" podUID="2f110c17-1134-4f5c-b9a6-51abb2ac481f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:20.770975 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:20.770087    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:21.161594 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.161496    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8" event={"ID":"c23f0ba4-8743-4fae-bdbe-307266213d9e","Type":"ContainerStarted","Data":"42559665b1365922fc2ffe941d3b5804a0dab2d915f23211a98ff92d352baef9"}
Nov 02 23:13:21.161883 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.161840    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8" event={"ID":"c23f0ba4-8743-4fae-bdbe-307266213d9e","Type":"ContainerStarted","Data":"920a056f5ba9b8c20376589a2c5ae05a8fafa7e8c5f2a83b1de689449699c78a"}
Nov 02 23:13:21.163506 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.163486    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8"
Nov 02 23:13:21.252796 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.251322    4294 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" podUID="845035ca-c55e-4e32-a19a-dc3617e3d638" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:21.277379 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.277342    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 23:13:21.278001 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.277951    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp" podUID="67334c5b-8d17-478e-8248-0762c0e1653c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 23:13:21.319013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.310923    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8" podStartSLOduration=3.31090357 podStartE2EDuration="3.31090357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:21.304341979 +0000 UTC m=+1240.872696769" watchObservedRunningTime="2026-11-02 23:13:21.31090357 +0000 UTC m=+1240.879258358"
Nov 02 23:13:21.349219 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.349167    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 23:13:21.349534 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.349490    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws" podUID="72b6e169-e4d8-4a5d-b5a2-88d4c424503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:21.594006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.583253    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb" podUID="aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:21.594006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.584094    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:13:21.658177 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.652090    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 02 23:13:21.681008 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.677098    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh" podUID="e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:21.735521 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.733083    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9" podUID="43e3d0d6-1a8e-4536-bc46-b5785124405a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:21.773148 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.773105    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 02 23:13:21.774267 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.774235    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf" podUID="2f110c17-1134-4f5c-b9a6-51abb2ac481f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 23:13:21.774760 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.774740    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:13:21.778008 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.774925    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:13:21.872163 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.872059    4294 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9" podUID="43e3d0d6-1a8e-4536-bc46-b5785124405a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:21.902005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.886079    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j" podUID="f483c007-731c-42e7-a7f2-9b7877fbba41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 02 23:13:21.902005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:21.895303    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:13:22.095050 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:22.085289    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" podUID="845035ca-c55e-4e32-a19a-dc3617e3d638" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:22.317012 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:22.302099    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 23:13:22.317012 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:22.302171    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp" podUID="67334c5b-8d17-478e-8248-0762c0e1653c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 23:13:22.539407 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:22.539362    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 23:13:22.539699 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:22.539655    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w" podUID="50895b59-7759-4a46-aa8f-a1d691c255de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:22.592879 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:22.590550    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb" podUID="aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 02 23:13:22.670010 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:22.668492    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:22.670010 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:22.668963    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:13:22.679184 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:22.679145    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh" podUID="e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:22.680734 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:22.680714    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:13:22.722175 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:22.722133    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 23:13:22.722507 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:22.722473    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff" podUID="51c49a39-5522-4cec-9246-7de8e59d4b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:22.734107 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:22.734066    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9" podUID="43e3d0d6-1a8e-4536-bc46-b5785124405a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:22.735437 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:22.735413    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:13:22.829100 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:22.829054    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:22.870525 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:22.829177    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf" podUID="2f110c17-1134-4f5c-b9a6-51abb2ac481f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 02 23:13:22.916034 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:22.903701    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j" podUID="f483c007-731c-42e7-a7f2-9b7877fbba41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:23.107122 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:23.107071    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" podUID="845035ca-c55e-4e32-a19a-dc3617e3d638" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:23.107789 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:23.107764    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:13:23.347057 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:23.347022    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 23:13:23.360120 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:23.360073    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp" podUID="67334c5b-8d17-478e-8248-0762c0e1653c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 23:13:23.574525 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:23.413210    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 23:13:23.574525 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:23.574468    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg" podUID="8a3538a4-c25e-4d3d-97e5-267a770bbb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:23.598184 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:23.598084    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb" podUID="aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:23.701005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:23.678096    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:23.701005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:23.681171    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh" podUID="e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:23.788790 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:23.788748    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9" podUID="43e3d0d6-1a8e-4536-bc46-b5785124405a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 23:13:23.883128 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:23.876199    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf" podUID="2f110c17-1134-4f5c-b9a6-51abb2ac481f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:23.883128 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:23.879714    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:23.925137 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:23.925090    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j" podUID="f483c007-731c-42e7-a7f2-9b7877fbba41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:24.131574 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:24.131518    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" podUID="845035ca-c55e-4e32-a19a-dc3617e3d638" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:24.308002 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:24.304210    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 23:13:24.308002 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:24.304276    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp" podUID="67334c5b-8d17-478e-8248-0762c0e1653c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 23:13:24.365934 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:24.363322    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:13:24.425991 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:24.425878    4294 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb" podUID="aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:24.882006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:24.811431    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh" podUID="e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:24.882006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:24.873506    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb" podUID="aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:24.925555 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:24.925511    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 02 23:13:24.986167 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:24.986114    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9" podUID="43e3d0d6-1a8e-4536-bc46-b5785124405a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:25.055221 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:25.055175    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j" podUID="f483c007-731c-42e7-a7f2-9b7877fbba41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:25.235060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:25.208004    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:25.520561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:25.132489    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" podUID="845035ca-c55e-4e32-a19a-dc3617e3d638" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:25.823899 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:25.823106    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh" podUID="e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:25.823899 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:25.823561    4294 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded"
Nov 02 23:13:26.086241 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:25.565397    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 23:13:26.086241 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.062881    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp" podUID="67334c5b-8d17-478e-8248-0762c0e1653c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 23:13:26.116229 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.088251    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9" podUID="43e3d0d6-1a8e-4536-bc46-b5785124405a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:26.116229 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.088764    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:26.154008 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.149241    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j" podUID="f483c007-731c-42e7-a7f2-9b7877fbba41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:26.211072 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.209380    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:26.282002 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.280579    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:26.282002 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.280680    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 02 23:13:26.282002 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.280721    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp" podUID="67334c5b-8d17-478e-8248-0762c0e1653c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 02 23:13:26.320000 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.314679    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:13:26.321189 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.321156    4294 generic.go:356] "Generic (PLEG): container finished" podID="96fc6247-0b37-40df-b795-2c799c981876" containerID="c0d05befac874f4db919144cda4523025522f5347d227f20168f0a4ff352e1dd" exitCode=0
Nov 02 23:13:26.359172 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:13:26.359094    4294 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.336s"
Nov 02 23:13:26.359172 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.359165    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:13:26.359394 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.359207    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:13:26.359394 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.359237    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 02 23:13:26.359394 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.359292    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl" event={"ID":"96fc6247-0b37-40df-b795-2c799c981876","Type":"ContainerDied","Data":"c0d05befac874f4db919144cda4523025522f5347d227f20168f0a4ff352e1dd"}
Nov 02 23:13:26.359394 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.359339    4294 scope.go:122] "RemoveContainer" containerID="c0d05befac874f4db919144cda4523025522f5347d227f20168f0a4ff352e1dd"
Nov 02 23:13:26.370018 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.362278    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:13:26.370457 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.370438    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:13:26.378997 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.375808    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:13:26.378997 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.376734    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:13:26.418385 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.418347    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-workload-certs\") pod \"96fc6247-0b37-40df-b795-2c799c981876\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") "
Nov 02 23:13:26.418598 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.418397    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96fc6247-0b37-40df-b795-2c799c981876-istio-token\" (UniqueName: \"kubernetes.io/projected/96fc6247-0b37-40df-b795-2c799c981876-istio-token\") pod \"96fc6247-0b37-40df-b795-2c799c981876\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") "
Nov 02 23:13:26.418598 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.418425    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-workload-socket\") pod \"96fc6247-0b37-40df-b795-2c799c981876\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") "
Nov 02 23:13:26.418598 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.418451    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96fc6247-0b37-40df-b795-2c799c981876-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96fc6247-0b37-40df-b795-2c799c981876-istiod-ca-cert\") pod \"96fc6247-0b37-40df-b795-2c799c981876\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") "
Nov 02 23:13:26.418598 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.418473    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96fc6247-0b37-40df-b795-2c799c981876-kube-api-access-f9qxz\" (UniqueName: \"kubernetes.io/projected/96fc6247-0b37-40df-b795-2c799c981876-kube-api-access-f9qxz\") pod \"96fc6247-0b37-40df-b795-2c799c981876\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") "
Nov 02 23:13:26.418598 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.418533    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96fc6247-0b37-40df-b795-2c799c981876-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96fc6247-0b37-40df-b795-2c799c981876-istio-podinfo\") pod \"96fc6247-0b37-40df-b795-2c799c981876\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") "
Nov 02 23:13:26.418598 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.418568    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-istio-data\") pod \"96fc6247-0b37-40df-b795-2c799c981876\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") "
Nov 02 23:13:26.418937 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.418634    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-istio-envoy\") pod \"96fc6247-0b37-40df-b795-2c799c981876\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") "
Nov 02 23:13:26.418937 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.418668    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-credential-socket\") pod \"96fc6247-0b37-40df-b795-2c799c981876\" (UID: \"96fc6247-0b37-40df-b795-2c799c981876\") "
Nov 02 23:13:26.421800 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.421769    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-workload-socket" pod "96fc6247-0b37-40df-b795-2c799c981876" (UID: "96fc6247-0b37-40df-b795-2c799c981876"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:13:26.422147 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.422117    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-workload-certs" pod "96fc6247-0b37-40df-b795-2c799c981876" (UID: "96fc6247-0b37-40df-b795-2c799c981876"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:13:26.422463 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.422441    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96fc6247-0b37-40df-b795-2c799c981876-istiod-ca-cert" pod "96fc6247-0b37-40df-b795-2c799c981876" (UID: "96fc6247-0b37-40df-b795-2c799c981876"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:13:26.423323 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.423302    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-credential-socket" pod "96fc6247-0b37-40df-b795-2c799c981876" (UID: "96fc6247-0b37-40df-b795-2c799c981876"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:13:26.423492 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.423466    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-istio-data" pod "96fc6247-0b37-40df-b795-2c799c981876" (UID: "96fc6247-0b37-40df-b795-2c799c981876"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:13:26.424403 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.424334    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96fc6247-0b37-40df-b795-2c799c981876-istio-token" pod "96fc6247-0b37-40df-b795-2c799c981876" (UID: "96fc6247-0b37-40df-b795-2c799c981876"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:13:26.424624 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.424594    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96fc6247-0b37-40df-b795-2c799c981876-istio-podinfo" pod "96fc6247-0b37-40df-b795-2c799c981876" (UID: "96fc6247-0b37-40df-b795-2c799c981876"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:13:26.426315 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.426290    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-istio-envoy" pod "96fc6247-0b37-40df-b795-2c799c981876" (UID: "96fc6247-0b37-40df-b795-2c799c981876"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:13:26.426965 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.426938    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96fc6247-0b37-40df-b795-2c799c981876-kube-api-access-f9qxz" pod "96fc6247-0b37-40df-b795-2c799c981876" (UID: "96fc6247-0b37-40df-b795-2c799c981876"). InnerVolumeSpecName "kube-api-access-f9qxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:13:26.454423 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.454355    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:26.522646 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.522404    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96fc6247-0b37-40df-b795-2c799c981876-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:13:26.522646 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.522439    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:13:26.522646 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.522457    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96fc6247-0b37-40df-b795-2c799c981876-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:13:26.522646 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.522474    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9qxz\" (UniqueName: \"kubernetes.io/projected/96fc6247-0b37-40df-b795-2c799c981876-kube-api-access-f9qxz\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:13:26.522646 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.522488    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96fc6247-0b37-40df-b795-2c799c981876-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:13:26.522646 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.522501    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:13:26.522646 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.522515    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:13:26.522646 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.522530    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:13:26.522646 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.522544    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96fc6247-0b37-40df-b795-2c799c981876-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:13:26.909232 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.909179    4294 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 23:13:26.909433 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:26.909271    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl" podUID="96fc6247-0b37-40df-b795-2c799c981876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:27.332776 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:27.332746    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"
Nov 02 23:13:27.333009 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:27.332745    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl" event={"ID":"96fc6247-0b37-40df-b795-2c799c981876","Type":"ContainerDied","Data":"51b08f87026af2f02059867c823694c1aff9b977eb1c648dab2c1abb4cc3bf80"}
Nov 02 23:13:27.498407 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:27.498334    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"]
Nov 02 23:13:27.505069 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:27.505031    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-lgcxl"]
Nov 02 23:13:27.754399 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:27.754362    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:27.754958 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:27.754935    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:27.756109 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:27.756082    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:13:28.366948 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:28.366916    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:28.549758 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:28.549725    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96fc6247-0b37-40df-b795-2c799c981876" path="/var/lib/kubelet/pods/96fc6247-0b37-40df-b795-2c799c981876/volumes"
Nov 02 23:13:30.438478 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:30.438443    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:33.432502 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:33.432471    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8"
Nov 02 23:13:35.065433 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:35.065399    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:39.642074 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:39.642042    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:40.084371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.084339    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"]
Nov 02 23:13:40.084950 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.084936    4294 cpu_manager.go:414] "containerMap: removing container" podUID="96fc6247-0b37-40df-b795-2c799c981876" containerName="istio-proxy"
Nov 02 23:13:40.085011 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.084951    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96fc6247-0b37-40df-b795-2c799c981876" containerName="istio-proxy"
Nov 02 23:13:40.085067 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.085057    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="96fc6247-0b37-40df-b795-2c799c981876" containerName="istio-proxy"
Nov 02 23:13:40.088502 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.088486    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.146567 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.146526    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"]
Nov 02 23:13:40.186791 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.186760    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 23:13:40.186791 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.186760    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 23:13:40.187016 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.186773    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 23:13:40.187016 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.186817    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tbzbd\""
Nov 02 23:13:40.239714 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.239679    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.239884 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.239864    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5k2n\" (UniqueName: \"kubernetes.io/projected/28a17c69-bf9c-4920-b804-429af4fbfadf-kube-api-access-t5k2n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.239943 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.239924    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.240069 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.240050    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.240148 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.240109    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a17c69-bf9c-4920-b804-429af4fbfadf-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.240213 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.240193    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.240250 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.240229    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.341386 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.341310    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.341569 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.341472    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5k2n\" (UniqueName: \"kubernetes.io/projected/28a17c69-bf9c-4920-b804-429af4fbfadf-kube-api-access-t5k2n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.341569 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.341544    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.341693 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.341585    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.341693 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.341637    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a17c69-bf9c-4920-b804-429af4fbfadf-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.341693 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.341679    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.341844 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.341712    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.342484 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.342453    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.343815 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.343797    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.343940 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.343917    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a17c69-bf9c-4920-b804-429af4fbfadf-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.344086 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.344069    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.344163 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.344146    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.396194 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.396166    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.396335 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.396313    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5k2n\" (UniqueName: \"kubernetes.io/projected/28a17c69-bf9c-4920-b804-429af4fbfadf-kube-api-access-t5k2n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hmbx6\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.401176 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.401154    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:40.819898 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.819870    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"]
Nov 02 23:13:40.820547 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:13:40.820515    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a17c69_bf9c_4920_b804_429af4fbfadf.slice/crio-6d0c1b56fd41b414399f5a8ff883fb73001d2ac479f391f8e9488250ff63a933 WatchSource:0}: Error finding container 6d0c1b56fd41b414399f5a8ff883fb73001d2ac479f391f8e9488250ff63a933: Status 404 returned error can't find the container with id 6d0c1b56fd41b414399f5a8ff883fb73001d2ac479f391f8e9488250ff63a933
Nov 02 23:13:40.823741 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.823703    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:13:40.823826 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:40.823779    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:13:41.484227 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:41.484192    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6" event={"ID":"28a17c69-bf9c-4920-b804-429af4fbfadf","Type":"ContainerStarted","Data":"f76f5596d63b6c425ec113d94906be1680a9f5547cf6f7bec35345b0561f9d5b"}
Nov 02 23:13:41.484437 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:41.484241    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6" event={"ID":"28a17c69-bf9c-4920-b804-429af4fbfadf","Type":"ContainerStarted","Data":"6d0c1b56fd41b414399f5a8ff883fb73001d2ac479f391f8e9488250ff63a933"}
Nov 02 23:13:41.484514 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:41.484453    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:41.486234 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:41.486210    4294 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-hmbx6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 23:13:41.486373 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:41.486262    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6" podUID="28a17c69-bf9c-4920-b804-429af4fbfadf" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 23:13:41.609649 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:41.609592    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6" podStartSLOduration=1.609576914 podStartE2EDuration="1.609576914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:13:41.606888263 +0000 UTC m=+1261.175243050" watchObservedRunningTime="2026-11-02 23:13:41.609576914 +0000 UTC m=+1261.177931701"
Nov 02 23:13:41.752252 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:13:41.752209    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:13:42.491363 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:42.491333    4294 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-hmbx6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 23:13:42.491868 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:42.491385    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6" podUID="28a17c69-bf9c-4920-b804-429af4fbfadf" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 23:13:44.206421 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:44.206386    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:44.379861 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:44.379824    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:44.553029 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:44.552954    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:45.492255 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:45.492226    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:13:47.811005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:47.796719    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9" podUID="43e3d0d6-1a8e-4536-bc46-b5785124405a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:47.884112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:47.884058    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf" podUID="2f110c17-1134-4f5c-b9a6-51abb2ac481f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:48.148170 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:48.148063    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" podUID="845035ca-c55e-4e32-a19a-dc3617e3d638" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 23:13:48.155093 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:48.155051    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j" podUID="f483c007-731c-42e7-a7f2-9b7877fbba41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 02 23:13:48.624167 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:48.624117    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb" podUID="aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:48.697006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:48.696078    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:48.802746 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:48.801099    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9" podUID="43e3d0d6-1a8e-4536-bc46-b5785124405a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:48.802746 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:48.801827    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:13:48.837006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:48.836097    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh" podUID="e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:48.893183 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:48.893076    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:48.898845 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:48.898803    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf" podUID="2f110c17-1134-4f5c-b9a6-51abb2ac481f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:48.899434 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:48.899410    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:13:49.166246 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:49.166149    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j" podUID="f483c007-731c-42e7-a7f2-9b7877fbba41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:49.378005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:49.370450    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" podUID="845035ca-c55e-4e32-a19a-dc3617e3d638" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded"
Nov 02 23:13:49.475197 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:49.428752    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:49.475804 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:49.438051    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:13:49.476091 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:49.476073    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:13:49.757113 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:49.700842    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:49.879005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:49.807536    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9" podUID="43e3d0d6-1a8e-4536-bc46-b5785124405a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Nov 02 23:13:49.910602 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:49.875109    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh" podUID="e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded"
Nov 02 23:13:49.910602 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:49.886461    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:13:49.910602 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:49.886554    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:13:49.955036 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:49.937618    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf" podUID="2f110c17-1134-4f5c-b9a6-51abb2ac481f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:49.956023 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:49.955930    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:49.957398 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:49.957377    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:13:50.044937 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:50.044856    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:50.129571 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:50.129530    4294 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j" podUID="f483c007-731c-42e7-a7f2-9b7877fbba41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": context deadline exceeded"
Nov 02 23:13:50.185020 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:50.174507    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j" podUID="f483c007-731c-42e7-a7f2-9b7877fbba41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 02 23:13:50.393052 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:50.392915    4294 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh" podUID="e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:50.506418 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:50.506376    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" podUID="845035ca-c55e-4e32-a19a-dc3617e3d638" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:50.507513 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:50.506928    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:50.730169 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:50.730087    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 02 23:13:50.741517 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:50.741247    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 02 23:13:50.744803 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:50.744760    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 02 23:13:50.824967 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:50.824937    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 02 23:13:50.991351 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:50.991306    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:51.114181 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:51.114130    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:13:51.176140 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:51.176094    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j" podUID="f483c007-731c-42e7-a7f2-9b7877fbba41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 02 23:13:51.235011 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:51.228795    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:51.286748 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:51.286076    4294 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" podUID="845035ca-c55e-4e32-a19a-dc3617e3d638" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded"
Nov 02 23:13:51.367004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:51.363461    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 02 23:13:51.412083 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:51.401139    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j"
Nov 02 23:13:51.548880 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:51.548790    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:51.818709 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:51.818610    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 02 23:13:55.677800 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:55.677758    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:55.857349 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:55.857317    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:13:56.136279 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:13:56.136242    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:01.071554 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:01.071518    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:02.223299 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:02.223262    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:02.409143 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:02.409106    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:04.598028 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:04.597994    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:04.828335 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:04.828298    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:05.755711 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:05.755668    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:05.966150 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:05.966115    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:10.584286 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:10.584248    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:11.754973 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:14:11.754923    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:14:15.384331 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:15.384300    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:20.379599 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:20.379559    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:24.289772 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:24.289666    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:24.612659 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:24.612510    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:25.379917 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:25.379877    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:28.646908 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:28.646871    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:29.560162 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:29.559673    4294 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": read tcp 10.133.0.2:41668->10.133.0.85:15021: read: connection reset by peer" start-of-body=
Nov 02 23:14:29.560162 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:29.559775    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn" podUID="d6735e93-95bc-4514-8739-31725bf4fd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": read tcp 10.133.0.2:41668->10.133.0.85:15021: read: connection reset by peer"
Nov 02 23:14:30.474214 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:30.474178    4294 generic.go:356] "Generic (PLEG): container finished" podID="d6735e93-95bc-4514-8739-31725bf4fd5d" containerID="94bddf21811a60a483efc68980b67ed91f0d4b84103c92251a212498aad9ba00" exitCode=0
Nov 02 23:14:30.474715 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:30.474207    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn" event={"ID":"d6735e93-95bc-4514-8739-31725bf4fd5d","Type":"ContainerDied","Data":"94bddf21811a60a483efc68980b67ed91f0d4b84103c92251a212498aad9ba00"}
Nov 02 23:14:30.474801 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:30.474783    4294 scope.go:122] "RemoveContainer" containerID="94bddf21811a60a483efc68980b67ed91f0d4b84103c92251a212498aad9ba00"
Nov 02 23:14:30.476494 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:30.476464    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:14:30.476629 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:30.476545    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:14:30.476629 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:30.476594    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:14:30.555036 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:30.555003    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:30.739234 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:30.739143    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:30.947017 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:30.946966    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:31.486901 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:31.486866    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn" event={"ID":"d6735e93-95bc-4514-8739-31725bf4fd5d","Type":"ContainerStarted","Data":"37c4fcd5bec4b7567570b4d4653ce7d9d1ce25162863df8b420a33e3c0b22771"}
Nov 02 23:14:31.875003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:31.874403    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:14:31.875003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:31.874538    4294 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 02 23:14:31.875003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:31.874583    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn" podUID="d6735e93-95bc-4514-8739-31725bf4fd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 02 23:14:32.878169 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:32.878114    4294 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 02 23:14:32.878683 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:32.878207    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn" podUID="d6735e93-95bc-4514-8739-31725bf4fd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 02 23:14:33.874113 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:33.874074    4294 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 02 23:14:33.874415 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:33.874383    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn" podUID="d6735e93-95bc-4514-8739-31725bf4fd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 02 23:14:34.872467 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:34.872424    4294 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 02 23:14:34.873124 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:34.873088    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn" podUID="d6735e93-95bc-4514-8739-31725bf4fd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 02 23:14:35.866408 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:35.866369    4294 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 02 23:14:35.867552 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:35.867516    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn" podUID="d6735e93-95bc-4514-8739-31725bf4fd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 02 23:14:35.969221 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:35.969186    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:36.865658 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:36.865568    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:14:36.866110 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:36.866085    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:14:36.866730 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:36.866712    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:14:40.798916 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:40.798881    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:41.237030 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:41.236922    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:41.751390 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:14:41.751348    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:14:45.832746 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:45.832711    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:50.795527 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:50.795489    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:14:56.061495 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:14:56.061461    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:00.175400 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.175352    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82"]
Nov 02 23:15:00.178783 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.178759    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82"
Nov 02 23:15:00.198404 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.198374    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8"]
Nov 02 23:15:00.201950 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.201912    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8"
Nov 02 23:15:00.202371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.202347    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p"]
Nov 02 23:15:00.205516 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.205498    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p"
Nov 02 23:15:00.242178 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.242155    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\""
Nov 02 23:15:00.264173 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.264143    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82"]
Nov 02 23:15:00.302446 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.302415    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dc2z\" (UniqueName: \"kubernetes.io/projected/e08c76d9-473c-4282-b75b-87e625e8c9b5-kube-api-access-6dc2z\") pod \"maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p\" (UID: \"e08c76d9-473c-4282-b75b-87e625e8c9b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p"
Nov 02 23:15:00.302608 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.302465    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc8bv\" (UniqueName: \"kubernetes.io/projected/b40d025d-9690-4dcb-88b0-2276f3327ae2-kube-api-access-pc8bv\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82\" (UID: \"b40d025d-9690-4dcb-88b0-2276f3327ae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82"
Nov 02 23:15:00.302608 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.302528    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8fd2\" (UniqueName: \"kubernetes.io/projected/6b5888be-67e7-4234-b17d-767ca7bdc1bc-kube-api-access-x8fd2\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8\" (UID: \"6b5888be-67e7-4234-b17d-767ca7bdc1bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8"
Nov 02 23:15:00.302608 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.302578    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b5888be-67e7-4234-b17d-767ca7bdc1bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8\" (UID: \"6b5888be-67e7-4234-b17d-767ca7bdc1bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8"
Nov 02 23:15:00.302772 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.302724    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40d025d-9690-4dcb-88b0-2276f3327ae2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82\" (UID: \"b40d025d-9690-4dcb-88b0-2276f3327ae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82"
Nov 02 23:15:00.302870 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.302849    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08c76d9-473c-4282-b75b-87e625e8c9b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p\" (UID: \"e08c76d9-473c-4282-b75b-87e625e8c9b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p"
Nov 02 23:15:00.309144 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.309115    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8"]
Nov 02 23:15:00.369284 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.369255    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p"]
Nov 02 23:15:00.405386 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.404197    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40d025d-9690-4dcb-88b0-2276f3327ae2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82\" (UID: \"b40d025d-9690-4dcb-88b0-2276f3327ae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82"
Nov 02 23:15:00.405386 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.404305    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08c76d9-473c-4282-b75b-87e625e8c9b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p\" (UID: \"e08c76d9-473c-4282-b75b-87e625e8c9b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p"
Nov 02 23:15:00.405386 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.404473    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dc2z\" (UniqueName: \"kubernetes.io/projected/e08c76d9-473c-4282-b75b-87e625e8c9b5-kube-api-access-6dc2z\") pod \"maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p\" (UID: \"e08c76d9-473c-4282-b75b-87e625e8c9b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p"
Nov 02 23:15:00.405386 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.404516    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc8bv\" (UniqueName: \"kubernetes.io/projected/b40d025d-9690-4dcb-88b0-2276f3327ae2-kube-api-access-pc8bv\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82\" (UID: \"b40d025d-9690-4dcb-88b0-2276f3327ae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82"
Nov 02 23:15:00.405386 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.404545    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8fd2\" (UniqueName: \"kubernetes.io/projected/6b5888be-67e7-4234-b17d-767ca7bdc1bc-kube-api-access-x8fd2\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8\" (UID: \"6b5888be-67e7-4234-b17d-767ca7bdc1bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8"
Nov 02 23:15:00.405386 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.404589    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b5888be-67e7-4234-b17d-767ca7bdc1bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8\" (UID: \"6b5888be-67e7-4234-b17d-767ca7bdc1bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8"
Nov 02 23:15:00.405386 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.405117    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08c76d9-473c-4282-b75b-87e625e8c9b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p\" (UID: \"e08c76d9-473c-4282-b75b-87e625e8c9b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p"
Nov 02 23:15:00.405386 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.405121    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40d025d-9690-4dcb-88b0-2276f3327ae2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82\" (UID: \"b40d025d-9690-4dcb-88b0-2276f3327ae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82"
Nov 02 23:15:00.405386 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.405337    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b5888be-67e7-4234-b17d-767ca7bdc1bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8\" (UID: \"6b5888be-67e7-4234-b17d-767ca7bdc1bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8"
Nov 02 23:15:00.459365 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.459300    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc8bv\" (UniqueName: \"kubernetes.io/projected/b40d025d-9690-4dcb-88b0-2276f3327ae2-kube-api-access-pc8bv\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82\" (UID: \"b40d025d-9690-4dcb-88b0-2276f3327ae2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82"
Nov 02 23:15:00.459497 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.459432    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8fd2\" (UniqueName: \"kubernetes.io/projected/6b5888be-67e7-4234-b17d-767ca7bdc1bc-kube-api-access-x8fd2\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8\" (UID: \"6b5888be-67e7-4234-b17d-767ca7bdc1bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8"
Nov 02 23:15:00.459605 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.459584    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dc2z\" (UniqueName: \"kubernetes.io/projected/e08c76d9-473c-4282-b75b-87e625e8c9b5-kube-api-access-6dc2z\") pod \"maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p\" (UID: \"e08c76d9-473c-4282-b75b-87e625e8c9b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p"
Nov 02 23:15:00.491254 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.491221    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82"
Nov 02 23:15:00.516045 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.516014    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8"
Nov 02 23:15:00.522567 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.522540    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p"
Nov 02 23:15:00.739414 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.739333    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:00.862162 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:15:00.862130    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40d025d_9690_4dcb_88b0_2276f3327ae2.slice/crio-b2934030ccdba8f9cdb835c602fcbfe8e74bff7be49d98e08b948ccc027bdac9 WatchSource:0}: Error finding container b2934030ccdba8f9cdb835c602fcbfe8e74bff7be49d98e08b948ccc027bdac9: Status 404 returned error can't find the container with id b2934030ccdba8f9cdb835c602fcbfe8e74bff7be49d98e08b948ccc027bdac9
Nov 02 23:15:00.862414 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.862393    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82"]
Nov 02 23:15:00.864681 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.864660    4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 23:15:00.909846 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:15:00.909820    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5888be_67e7_4234_b17d_767ca7bdc1bc.slice/crio-868e1b006c031e77afcb61a5203afb23ad60d34fa926b342a22278507b9c61c3 WatchSource:0}: Error finding container 868e1b006c031e77afcb61a5203afb23ad60d34fa926b342a22278507b9c61c3: Status 404 returned error can't find the container with id 868e1b006c031e77afcb61a5203afb23ad60d34fa926b342a22278507b9c61c3
Nov 02 23:15:00.910141 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.910121    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8"]
Nov 02 23:15:00.916842 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.916819    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:00.959420 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:00.959390    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p"]
Nov 02 23:15:00.959861 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:15:00.959834    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08c76d9_473c_4282_b75b_87e625e8c9b5.slice/crio-5cff2c64b0d243ded62a826d0b3e2e1a42a696b44306293d687798f3a52eec3b WatchSource:0}: Error finding container 5cff2c64b0d243ded62a826d0b3e2e1a42a696b44306293d687798f3a52eec3b: Status 404 returned error can't find the container with id 5cff2c64b0d243ded62a826d0b3e2e1a42a696b44306293d687798f3a52eec3b
Nov 02 23:15:01.676169 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:01.676130    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8" event={"ID":"6b5888be-67e7-4234-b17d-767ca7bdc1bc","Type":"ContainerStarted","Data":"868e1b006c031e77afcb61a5203afb23ad60d34fa926b342a22278507b9c61c3"}
Nov 02 23:15:01.677420 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:01.677383    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p" event={"ID":"e08c76d9-473c-4282-b75b-87e625e8c9b5","Type":"ContainerStarted","Data":"5cff2c64b0d243ded62a826d0b3e2e1a42a696b44306293d687798f3a52eec3b"}
Nov 02 23:15:01.678547 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:01.678515    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82" event={"ID":"b40d025d-9690-4dcb-88b0-2276f3327ae2","Type":"ContainerStarted","Data":"b2934030ccdba8f9cdb835c602fcbfe8e74bff7be49d98e08b948ccc027bdac9"}
Nov 02 23:15:05.921189 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:05.921148    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:06.727621 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:06.727519    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p" event={"ID":"e08c76d9-473c-4282-b75b-87e625e8c9b5","Type":"ContainerStarted","Data":"cc2f28ca5736159b9e8126dee07c6d7bad4157759dfa96f8479450f8613fd31c"}
Nov 02 23:15:06.729217 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:06.729190    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82" event={"ID":"b40d025d-9690-4dcb-88b0-2276f3327ae2","Type":"ContainerStarted","Data":"df626be15cdd4e7288d521200be10accd5531ab1a953aad1eae7692b2bf76bd3"}
Nov 02 23:15:06.730764 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:06.730742    4294 generic.go:356] "Generic (PLEG): container finished" podID="6b5888be-67e7-4234-b17d-767ca7bdc1bc" containerID="49f11064bf5298faf3e43fca85c4fb754a68a5653fa588950d3222f3311b8e10" exitCode=60
Nov 02 23:15:06.730764 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:06.730769    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8" event={"ID":"6b5888be-67e7-4234-b17d-767ca7bdc1bc","Type":"ContainerDied","Data":"49f11064bf5298faf3e43fca85c4fb754a68a5653fa588950d3222f3311b8e10"}
Nov 02 23:15:06.731049 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:06.731031    4294 scope.go:122] "RemoveContainer" containerID="49f11064bf5298faf3e43fca85c4fb754a68a5653fa588950d3222f3311b8e10"
Nov 02 23:15:06.887896 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:06.887838    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p" podStartSLOduration=1.9202433 podStartE2EDuration="6.887821519s" totalImagesPullingTime="4.967578219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:06.883859312 +0000 UTC m=+1346.452214101" watchObservedRunningTime="2026-11-02 23:15:06.887821519 +0000 UTC m=+1346.456176327"
Nov 02 23:15:06.942298 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:06.942254    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:06.996952 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:06.996906    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82" podStartSLOduration=1.93582276 podStartE2EDuration="6.996890279s" totalImagesPullingTime="5.061067519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:06.993552075 +0000 UTC m=+1346.561906864" watchObservedRunningTime="2026-11-02 23:15:06.996890279 +0000 UTC m=+1346.565245065"
Nov 02 23:15:07.745320 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:07.745271    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8" event={"ID":"6b5888be-67e7-4234-b17d-767ca7bdc1bc","Type":"ContainerStarted","Data":"08532073535ee63a2b4d0af31b806c60af465cea1814fb732b511419e87351aa"}
Nov 02 23:15:07.750759 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:07.750729    4294 generic.go:356] "Generic (PLEG): container finished" podID="e08c76d9-473c-4282-b75b-87e625e8c9b5" containerID="cc2f28ca5736159b9e8126dee07c6d7bad4157759dfa96f8479450f8613fd31c" exitCode=60
Nov 02 23:15:07.750940 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:07.750777    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p" event={"ID":"e08c76d9-473c-4282-b75b-87e625e8c9b5","Type":"ContainerDied","Data":"cc2f28ca5736159b9e8126dee07c6d7bad4157759dfa96f8479450f8613fd31c"}
Nov 02 23:15:07.751174 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:07.751140    4294 scope.go:122] "RemoveContainer" containerID="cc2f28ca5736159b9e8126dee07c6d7bad4157759dfa96f8479450f8613fd31c"
Nov 02 23:15:07.752663 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:07.752640    4294 generic.go:356] "Generic (PLEG): container finished" podID="b40d025d-9690-4dcb-88b0-2276f3327ae2" containerID="df626be15cdd4e7288d521200be10accd5531ab1a953aad1eae7692b2bf76bd3" exitCode=60
Nov 02 23:15:07.752824 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:07.752796    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82" event={"ID":"b40d025d-9690-4dcb-88b0-2276f3327ae2","Type":"ContainerDied","Data":"df626be15cdd4e7288d521200be10accd5531ab1a953aad1eae7692b2bf76bd3"}
Nov 02 23:15:07.752970 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:07.752948    4294 scope.go:122] "RemoveContainer" containerID="df626be15cdd4e7288d521200be10accd5531ab1a953aad1eae7692b2bf76bd3"
Nov 02 23:15:07.857560 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:07.857501    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8" podStartSLOduration=3.001782701 podStartE2EDuration="7.857483268s" totalImagesPullingTime="4.855700567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:15:07.853145406 +0000 UTC m=+1347.421500195" watchObservedRunningTime="2026-11-02 23:15:07.857483268 +0000 UTC m=+1347.425838056"
Nov 02 23:15:08.762628 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:08.762593    4294 generic.go:356] "Generic (PLEG): container finished" podID="6b5888be-67e7-4234-b17d-767ca7bdc1bc" containerID="08532073535ee63a2b4d0af31b806c60af465cea1814fb732b511419e87351aa" exitCode=60
Nov 02 23:15:08.763116 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:08.762672    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8" event={"ID":"6b5888be-67e7-4234-b17d-767ca7bdc1bc","Type":"ContainerDied","Data":"08532073535ee63a2b4d0af31b806c60af465cea1814fb732b511419e87351aa"}
Nov 02 23:15:08.763116 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:08.762745    4294 scope.go:122] "RemoveContainer" containerID="49f11064bf5298faf3e43fca85c4fb754a68a5653fa588950d3222f3311b8e10"
Nov 02 23:15:08.763116 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:08.763079    4294 scope.go:122] "RemoveContainer" containerID="08532073535ee63a2b4d0af31b806c60af465cea1814fb732b511419e87351aa"
Nov 02 23:15:08.763474 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:15:08.763446    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8_odh-ai-gateway-infra(6b5888be-67e7-4234-b17d-767ca7bdc1bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8" podUID="6b5888be-67e7-4234-b17d-767ca7bdc1bc"
Nov 02 23:15:08.764781 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:08.764756    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p" event={"ID":"e08c76d9-473c-4282-b75b-87e625e8c9b5","Type":"ContainerStarted","Data":"f8206598596ca6e219cf1fa2cff506f2fe339156bf9e38754dc90bd9d5b52c5a"}
Nov 02 23:15:08.766597 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:08.766574    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82" event={"ID":"b40d025d-9690-4dcb-88b0-2276f3327ae2","Type":"ContainerStarted","Data":"6154fd96c50237d1624158c4d58c91999865939a588ad9ef9b8c2f8cfded8fdb"}
Nov 02 23:15:09.237607 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:09.237508    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:09.775744 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:09.775717    4294 scope.go:122] "RemoveContainer" containerID="08532073535ee63a2b4d0af31b806c60af465cea1814fb732b511419e87351aa"
Nov 02 23:15:09.776227 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:15:09.776091    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8_odh-ai-gateway-infra(6b5888be-67e7-4234-b17d-767ca7bdc1bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8" podUID="6b5888be-67e7-4234-b17d-767ca7bdc1bc"
Nov 02 23:15:09.777432 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:09.777406    4294 generic.go:356] "Generic (PLEG): container finished" podID="e08c76d9-473c-4282-b75b-87e625e8c9b5" containerID="f8206598596ca6e219cf1fa2cff506f2fe339156bf9e38754dc90bd9d5b52c5a" exitCode=60
Nov 02 23:15:09.777572 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:09.777486    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p" event={"ID":"e08c76d9-473c-4282-b75b-87e625e8c9b5","Type":"ContainerDied","Data":"f8206598596ca6e219cf1fa2cff506f2fe339156bf9e38754dc90bd9d5b52c5a"}
Nov 02 23:15:09.777572 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:09.777541    4294 scope.go:122] "RemoveContainer" containerID="cc2f28ca5736159b9e8126dee07c6d7bad4157759dfa96f8479450f8613fd31c"
Nov 02 23:15:09.777767 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:09.777748    4294 scope.go:122] "RemoveContainer" containerID="f8206598596ca6e219cf1fa2cff506f2fe339156bf9e38754dc90bd9d5b52c5a"
Nov 02 23:15:09.778144 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:15:09.778102    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p_odh-ai-gateway-infra(e08c76d9-473c-4282-b75b-87e625e8c9b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p" podUID="e08c76d9-473c-4282-b75b-87e625e8c9b5"
Nov 02 23:15:09.779454 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:09.779435    4294 generic.go:356] "Generic (PLEG): container finished" podID="b40d025d-9690-4dcb-88b0-2276f3327ae2" containerID="6154fd96c50237d1624158c4d58c91999865939a588ad9ef9b8c2f8cfded8fdb" exitCode=60
Nov 02 23:15:09.779533 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:09.779476    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82" event={"ID":"b40d025d-9690-4dcb-88b0-2276f3327ae2","Type":"ContainerDied","Data":"6154fd96c50237d1624158c4d58c91999865939a588ad9ef9b8c2f8cfded8fdb"}
Nov 02 23:15:09.779738 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:09.779725    4294 scope.go:122] "RemoveContainer" containerID="6154fd96c50237d1624158c4d58c91999865939a588ad9ef9b8c2f8cfded8fdb"
Nov 02 23:15:09.780022 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:15:09.780001    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82_odh-ai-gateway-infra(b40d025d-9690-4dcb-88b0-2276f3327ae2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82" podUID="b40d025d-9690-4dcb-88b0-2276f3327ae2"
Nov 02 23:15:09.789939 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:09.789919    4294 scope.go:122] "RemoveContainer" containerID="df626be15cdd4e7288d521200be10accd5531ab1a953aad1eae7692b2bf76bd3"
Nov 02 23:15:10.788798 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:10.788767    4294 scope.go:122] "RemoveContainer" containerID="f8206598596ca6e219cf1fa2cff506f2fe339156bf9e38754dc90bd9d5b52c5a"
Nov 02 23:15:10.789361 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:15:10.789120    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p_odh-ai-gateway-infra(e08c76d9-473c-4282-b75b-87e625e8c9b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p" podUID="e08c76d9-473c-4282-b75b-87e625e8c9b5"
Nov 02 23:15:10.790514 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:10.790494    4294 scope.go:122] "RemoveContainer" containerID="6154fd96c50237d1624158c4d58c91999865939a588ad9ef9b8c2f8cfded8fdb"
Nov 02 23:15:10.790799 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:15:10.790777    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82_odh-ai-gateway-infra(b40d025d-9690-4dcb-88b0-2276f3327ae2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82" podUID="b40d025d-9690-4dcb-88b0-2276f3327ae2"
Nov 02 23:15:10.793165 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:10.793144    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:11.751609 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:15:11.751563    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:15:17.800796 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:17.800759    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:17.984173 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:17.984133    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:18.584203 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:18.584167    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:18.756544 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:18.756506    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:20.548288 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:20.548252    4294 scope.go:122] "RemoveContainer" containerID="6154fd96c50237d1624158c4d58c91999865939a588ad9ef9b8c2f8cfded8fdb"
Nov 02 23:15:20.552083 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:20.552062    4294 scope.go:122] "RemoveContainer" containerID="08532073535ee63a2b4d0af31b806c60af465cea1814fb732b511419e87351aa"
Nov 02 23:15:21.244758 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:21.244720    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:21.990264 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:21.990226    4294 generic.go:356] "Generic (PLEG): container finished" podID="6b5888be-67e7-4234-b17d-767ca7bdc1bc" containerID="4cbb383fafa63fd544b5603d99fe16baf3bc652453092cc5a05e4029c53c9d74" exitCode=60
Nov 02 23:15:21.990710 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:21.990298    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8" event={"ID":"6b5888be-67e7-4234-b17d-767ca7bdc1bc","Type":"ContainerDied","Data":"4cbb383fafa63fd544b5603d99fe16baf3bc652453092cc5a05e4029c53c9d74"}
Nov 02 23:15:21.990710 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:21.990375    4294 scope.go:122] "RemoveContainer" containerID="08532073535ee63a2b4d0af31b806c60af465cea1814fb732b511419e87351aa"
Nov 02 23:15:21.990857 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:21.990839    4294 scope.go:122] "RemoveContainer" containerID="4cbb383fafa63fd544b5603d99fe16baf3bc652453092cc5a05e4029c53c9d74"
Nov 02 23:15:21.991294 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:15:21.991264    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8_odh-ai-gateway-infra(6b5888be-67e7-4234-b17d-767ca7bdc1bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8" podUID="6b5888be-67e7-4234-b17d-767ca7bdc1bc"
Nov 02 23:15:21.995578 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:21.995238    4294 generic.go:356] "Generic (PLEG): container finished" podID="b40d025d-9690-4dcb-88b0-2276f3327ae2" containerID="c2b854f8b5758187228caa49b03a9a0196a0ea89a9a9da573bb92bd756f587c4" exitCode=60
Nov 02 23:15:21.995578 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:21.995323    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82" event={"ID":"b40d025d-9690-4dcb-88b0-2276f3327ae2","Type":"ContainerDied","Data":"c2b854f8b5758187228caa49b03a9a0196a0ea89a9a9da573bb92bd756f587c4"}
Nov 02 23:15:21.995813 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:21.995789    4294 scope.go:122] "RemoveContainer" containerID="c2b854f8b5758187228caa49b03a9a0196a0ea89a9a9da573bb92bd756f587c4"
Nov 02 23:15:21.996142 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:15:21.996115    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82_odh-ai-gateway-infra(b40d025d-9690-4dcb-88b0-2276f3327ae2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82" podUID="b40d025d-9690-4dcb-88b0-2276f3327ae2"
Nov 02 23:15:22.006866 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:22.006836    4294 scope.go:122] "RemoveContainer" containerID="6154fd96c50237d1624158c4d58c91999865939a588ad9ef9b8c2f8cfded8fdb"
Nov 02 23:15:22.630001 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:22.623893    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:22.880065 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:22.858476    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:23.129007 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:23.128067    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8"]
Nov 02 23:15:23.240677 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:23.239967    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82"]
Nov 02 23:15:23.548922 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:23.546478    4294 scope.go:122] "RemoveContainer" containerID="f8206598596ca6e219cf1fa2cff506f2fe339156bf9e38754dc90bd9d5b52c5a"
Nov 02 23:15:23.922496 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:23.922189    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8"
Nov 02 23:15:23.966686 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:23.966654    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82"
Nov 02 23:15:24.024445 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.024406    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b40d025d-9690-4dcb-88b0-2276f3327ae2-kube-api-access-pc8bv\" (UniqueName: \"kubernetes.io/projected/b40d025d-9690-4dcb-88b0-2276f3327ae2-kube-api-access-pc8bv\") pod \"b40d025d-9690-4dcb-88b0-2276f3327ae2\" (UID: \"b40d025d-9690-4dcb-88b0-2276f3327ae2\") "
Nov 02 23:15:24.024646 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.024551    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b40d025d-9690-4dcb-88b0-2276f3327ae2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40d025d-9690-4dcb-88b0-2276f3327ae2-maas-api-service-ca\") pod \"b40d025d-9690-4dcb-88b0-2276f3327ae2\" (UID: \"b40d025d-9690-4dcb-88b0-2276f3327ae2\") "
Nov 02 23:15:24.024646 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.024614    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b5888be-67e7-4234-b17d-767ca7bdc1bc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b5888be-67e7-4234-b17d-767ca7bdc1bc-maas-api-service-ca\") pod \"6b5888be-67e7-4234-b17d-767ca7bdc1bc\" (UID: \"6b5888be-67e7-4234-b17d-767ca7bdc1bc\") "
Nov 02 23:15:24.024752 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.024689    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b5888be-67e7-4234-b17d-767ca7bdc1bc-kube-api-access-x8fd2\" (UniqueName: \"kubernetes.io/projected/6b5888be-67e7-4234-b17d-767ca7bdc1bc-kube-api-access-x8fd2\") pod \"6b5888be-67e7-4234-b17d-767ca7bdc1bc\" (UID: \"6b5888be-67e7-4234-b17d-767ca7bdc1bc\") "
Nov 02 23:15:24.025781 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.025751    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b5888be-67e7-4234-b17d-767ca7bdc1bc-maas-api-service-ca" pod "6b5888be-67e7-4234-b17d-767ca7bdc1bc" (UID: "6b5888be-67e7-4234-b17d-767ca7bdc1bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:15:24.025957 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.025761    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40d025d-9690-4dcb-88b0-2276f3327ae2-maas-api-service-ca" pod "b40d025d-9690-4dcb-88b0-2276f3327ae2" (UID: "b40d025d-9690-4dcb-88b0-2276f3327ae2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:15:24.027969 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.027936    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b5888be-67e7-4234-b17d-767ca7bdc1bc-kube-api-access-x8fd2" pod "6b5888be-67e7-4234-b17d-767ca7bdc1bc" (UID: "6b5888be-67e7-4234-b17d-767ca7bdc1bc"). InnerVolumeSpecName "kube-api-access-x8fd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:15:24.029955 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.029932    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40d025d-9690-4dcb-88b0-2276f3327ae2-kube-api-access-pc8bv" pod "b40d025d-9690-4dcb-88b0-2276f3327ae2" (UID: "b40d025d-9690-4dcb-88b0-2276f3327ae2"). InnerVolumeSpecName "kube-api-access-pc8bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:15:24.067722 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.067686    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8" event={"ID":"6b5888be-67e7-4234-b17d-767ca7bdc1bc","Type":"ContainerDied","Data":"868e1b006c031e77afcb61a5203afb23ad60d34fa926b342a22278507b9c61c3"}
Nov 02 23:15:24.067913 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.067757    4294 scope.go:122] "RemoveContainer" containerID="4cbb383fafa63fd544b5603d99fe16baf3bc652453092cc5a05e4029c53c9d74"
Nov 02 23:15:24.067913 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.067704    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8"
Nov 02 23:15:24.071400 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.071370    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82" event={"ID":"b40d025d-9690-4dcb-88b0-2276f3327ae2","Type":"ContainerDied","Data":"b2934030ccdba8f9cdb835c602fcbfe8e74bff7be49d98e08b948ccc027bdac9"}
Nov 02 23:15:24.071507 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.071472    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82"
Nov 02 23:15:24.100826 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.100796    4294 scope.go:122] "RemoveContainer" containerID="c2b854f8b5758187228caa49b03a9a0196a0ea89a9a9da573bb92bd756f587c4"
Nov 02 23:15:24.126335 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.126300    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40d025d-9690-4dcb-88b0-2276f3327ae2-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:15:24.126335 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.126336    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b5888be-67e7-4234-b17d-767ca7bdc1bc-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:15:24.126517 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.126353    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8fd2\" (UniqueName: \"kubernetes.io/projected/6b5888be-67e7-4234-b17d-767ca7bdc1bc-kube-api-access-x8fd2\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:15:24.126517 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.126369    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc8bv\" (UniqueName: \"kubernetes.io/projected/b40d025d-9690-4dcb-88b0-2276f3327ae2-kube-api-access-pc8bv\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:15:24.146225 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:15:24.146144    4294 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5888be_67e7_4234_b17d_767ca7bdc1bc.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40d025d_9690_4dcb_88b0_2276f3327ae2.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 23:15:24.271233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.271189    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82"]
Nov 02 23:15:24.277688 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.277657    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894355-wdm82"]
Nov 02 23:15:24.401804 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.401717    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8"]
Nov 02 23:15:24.422406 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.422376    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894355-tj7g8"]
Nov 02 23:15:24.476243 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.476210    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:24.548947 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.548917    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b5888be-67e7-4234-b17d-767ca7bdc1bc" path="/var/lib/kubelet/pods/6b5888be-67e7-4234-b17d-767ca7bdc1bc/volumes"
Nov 02 23:15:24.549462 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.549446    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b40d025d-9690-4dcb-88b0-2276f3327ae2" path="/var/lib/kubelet/pods/b40d025d-9690-4dcb-88b0-2276f3327ae2/volumes"
Nov 02 23:15:24.705642 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:24.705553    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:25.080993 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:25.080950    4294 generic.go:356] "Generic (PLEG): container finished" podID="e08c76d9-473c-4282-b75b-87e625e8c9b5" containerID="9e3ab8bbb6caff4ef20a8ade4f31c45ee057e4fe34bbc4d7e363fceb8b7b8303" exitCode=60
Nov 02 23:15:25.081224 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:25.081026    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p" event={"ID":"e08c76d9-473c-4282-b75b-87e625e8c9b5","Type":"ContainerDied","Data":"9e3ab8bbb6caff4ef20a8ade4f31c45ee057e4fe34bbc4d7e363fceb8b7b8303"}
Nov 02 23:15:25.081224 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:25.081082    4294 scope.go:122] "RemoveContainer" containerID="f8206598596ca6e219cf1fa2cff506f2fe339156bf9e38754dc90bd9d5b52c5a"
Nov 02 23:15:25.081545 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:25.081526    4294 scope.go:122] "RemoveContainer" containerID="9e3ab8bbb6caff4ef20a8ade4f31c45ee057e4fe34bbc4d7e363fceb8b7b8303"
Nov 02 23:15:25.081916 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:15:25.081885    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p_odh-ai-gateway-infra(e08c76d9-473c-4282-b75b-87e625e8c9b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p" podUID="e08c76d9-473c-4282-b75b-87e625e8c9b5"
Nov 02 23:15:25.953429 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:25.951929    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:26.175004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:26.167265    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:26.205257 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:26.205168    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p"]
Nov 02 23:15:26.394776 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:26.394736    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p"
Nov 02 23:15:26.481502 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:26.481414    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e08c76d9-473c-4282-b75b-87e625e8c9b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08c76d9-473c-4282-b75b-87e625e8c9b5-maas-api-service-ca\") pod \"e08c76d9-473c-4282-b75b-87e625e8c9b5\" (UID: \"e08c76d9-473c-4282-b75b-87e625e8c9b5\") "
Nov 02 23:15:26.482223 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:26.482199    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e08c76d9-473c-4282-b75b-87e625e8c9b5-kube-api-access-6dc2z\" (UniqueName: \"kubernetes.io/projected/e08c76d9-473c-4282-b75b-87e625e8c9b5-kube-api-access-6dc2z\") pod \"e08c76d9-473c-4282-b75b-87e625e8c9b5\" (UID: \"e08c76d9-473c-4282-b75b-87e625e8c9b5\") "
Nov 02 23:15:26.483458 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:26.482117    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08c76d9-473c-4282-b75b-87e625e8c9b5-maas-api-service-ca" pod "e08c76d9-473c-4282-b75b-87e625e8c9b5" (UID: "e08c76d9-473c-4282-b75b-87e625e8c9b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:15:26.488642 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:26.488608    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08c76d9-473c-4282-b75b-87e625e8c9b5-kube-api-access-6dc2z" pod "e08c76d9-473c-4282-b75b-87e625e8c9b5" (UID: "e08c76d9-473c-4282-b75b-87e625e8c9b5"). InnerVolumeSpecName "kube-api-access-6dc2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:15:26.584934 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:26.584895    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08c76d9-473c-4282-b75b-87e625e8c9b5-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:15:26.584934 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:26.584936    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dc2z\" (UniqueName: \"kubernetes.io/projected/e08c76d9-473c-4282-b75b-87e625e8c9b5-kube-api-access-6dc2z\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:15:26.703287 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:26.703228    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:27.107236 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:27.107203    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p"
Nov 02 23:15:27.107236 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:27.107226    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p" event={"ID":"e08c76d9-473c-4282-b75b-87e625e8c9b5","Type":"ContainerDied","Data":"5cff2c64b0d243ded62a826d0b3e2e1a42a696b44306293d687798f3a52eec3b"}
Nov 02 23:15:27.107729 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:27.107285    4294 scope.go:122] "RemoveContainer" containerID="9e3ab8bbb6caff4ef20a8ade4f31c45ee057e4fe34bbc4d7e363fceb8b7b8303"
Nov 02 23:15:27.270043 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:27.269998    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p"]
Nov 02 23:15:27.315378 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:27.315333    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-b2a689-29894355-zzm7p"]
Nov 02 23:15:28.551726 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:28.551689    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e08c76d9-473c-4282-b75b-87e625e8c9b5" path="/var/lib/kubelet/pods/e08c76d9-473c-4282-b75b-87e625e8c9b5/volumes"
Nov 02 23:15:28.985730 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:28.985642    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:31.663855 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:31.663811    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:36.612678 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:36.612643    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:41.010680 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:41.010648    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:41.751178 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:15:41.751139    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:15:41.990192 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:41.990159    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:44.690461 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:44.690427    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:46.996183 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:46.996146    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:52.002191 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:52.002152    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:53.728737 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:53.728653    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:53.891308 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:53.891276    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:15:57.006124 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:15:57.006065    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:01.894688 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:01.894646    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:02.090211 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:02.090176    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:02.274276 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:02.274242    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:07.120272 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:07.120235    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:11.753146 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:16:11.753106    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:16:11.909053 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:11.909019    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:12.214241 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:12.214158    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:12.820184 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:12.820142    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:13.484340 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:13.484293    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:17.187887 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:17.187851    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:17.361321 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:17.361274    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:17.654162 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:17.654131    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:19.289010 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.288956    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"]
Nov 02 23:16:19.289577 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289562    4294 cpu_manager.go:414] "containerMap: removing container" podUID="6b5888be-67e7-4234-b17d-767ca7bdc1bc" containerName="cleanup"
Nov 02 23:16:19.289623 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289578    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b5888be-67e7-4234-b17d-767ca7bdc1bc" containerName="cleanup"
Nov 02 23:16:19.289623 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289591    4294 cpu_manager.go:414] "containerMap: removing container" podUID="e08c76d9-473c-4282-b75b-87e625e8c9b5" containerName="cleanup"
Nov 02 23:16:19.289623 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289598    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08c76d9-473c-4282-b75b-87e625e8c9b5" containerName="cleanup"
Nov 02 23:16:19.289623 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289606    4294 cpu_manager.go:414] "containerMap: removing container" podUID="e08c76d9-473c-4282-b75b-87e625e8c9b5" containerName="cleanup"
Nov 02 23:16:19.289623 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289612    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08c76d9-473c-4282-b75b-87e625e8c9b5" containerName="cleanup"
Nov 02 23:16:19.289778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289629    4294 cpu_manager.go:414] "containerMap: removing container" podUID="b40d025d-9690-4dcb-88b0-2276f3327ae2" containerName="cleanup"
Nov 02 23:16:19.289778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289634    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b40d025d-9690-4dcb-88b0-2276f3327ae2" containerName="cleanup"
Nov 02 23:16:19.289778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289641    4294 cpu_manager.go:414] "containerMap: removing container" podUID="b40d025d-9690-4dcb-88b0-2276f3327ae2" containerName="cleanup"
Nov 02 23:16:19.289778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289645    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b40d025d-9690-4dcb-88b0-2276f3327ae2" containerName="cleanup"
Nov 02 23:16:19.289778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289652    4294 cpu_manager.go:414] "containerMap: removing container" podUID="e08c76d9-473c-4282-b75b-87e625e8c9b5" containerName="cleanup"
Nov 02 23:16:19.289778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289656    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08c76d9-473c-4282-b75b-87e625e8c9b5" containerName="cleanup"
Nov 02 23:16:19.289778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289663    4294 cpu_manager.go:414] "containerMap: removing container" podUID="b40d025d-9690-4dcb-88b0-2276f3327ae2" containerName="cleanup"
Nov 02 23:16:19.289778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289667    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b40d025d-9690-4dcb-88b0-2276f3327ae2" containerName="cleanup"
Nov 02 23:16:19.289778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289674    4294 cpu_manager.go:414] "containerMap: removing container" podUID="6b5888be-67e7-4234-b17d-767ca7bdc1bc" containerName="cleanup"
Nov 02 23:16:19.289778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289678    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b5888be-67e7-4234-b17d-767ca7bdc1bc" containerName="cleanup"
Nov 02 23:16:19.289778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289687    4294 cpu_manager.go:414] "containerMap: removing container" podUID="6b5888be-67e7-4234-b17d-767ca7bdc1bc" containerName="cleanup"
Nov 02 23:16:19.289778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289691    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b5888be-67e7-4234-b17d-767ca7bdc1bc" containerName="cleanup"
Nov 02 23:16:19.290220 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289786    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b5888be-67e7-4234-b17d-767ca7bdc1bc" containerName="cleanup"
Nov 02 23:16:19.290220 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289793    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e08c76d9-473c-4282-b75b-87e625e8c9b5" containerName="cleanup"
Nov 02 23:16:19.290220 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289798    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e08c76d9-473c-4282-b75b-87e625e8c9b5" containerName="cleanup"
Nov 02 23:16:19.290220 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289808    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="b40d025d-9690-4dcb-88b0-2276f3327ae2" containerName="cleanup"
Nov 02 23:16:19.290220 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289813    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="b40d025d-9690-4dcb-88b0-2276f3327ae2" containerName="cleanup"
Nov 02 23:16:19.290220 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289820    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="b40d025d-9690-4dcb-88b0-2276f3327ae2" containerName="cleanup"
Nov 02 23:16:19.290220 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289825    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b5888be-67e7-4234-b17d-767ca7bdc1bc" containerName="cleanup"
Nov 02 23:16:19.290220 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.289831    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b5888be-67e7-4234-b17d-767ca7bdc1bc" containerName="cleanup"
Nov 02 23:16:19.292209 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.292194    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.399271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.399236    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-14b321-openshift-default-dockercfg-ssm6b\""
Nov 02 23:16:19.403206 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.403179    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"]
Nov 02 23:16:19.451721 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.451686    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-credential-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.451897 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.451733    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/602de6ee-5748-44d1-b389-7124d80e521b-istio-podinfo\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.451897 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.451824    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-workload-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.451972 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.451955    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqxp5\" (UniqueName: \"kubernetes.io/projected/602de6ee-5748-44d1-b389-7124d80e521b-kube-api-access-tqxp5\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.452075 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.452061    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-workload-certs\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.452110 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.452094    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-istio-data\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.452192 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.452177    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/602de6ee-5748-44d1-b389-7124d80e521b-istiod-ca-cert\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.452241 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.452230    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/602de6ee-5748-44d1-b389-7124d80e521b-istio-token\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.452279 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.452251    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-istio-envoy\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.553948 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.553831    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-workload-certs\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.553948 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.553887    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-istio-data\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.553948 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.553930    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/602de6ee-5748-44d1-b389-7124d80e521b-istiod-ca-cert\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.554268 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.554018    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/602de6ee-5748-44d1-b389-7124d80e521b-istio-token\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.554268 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.554047    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-istio-envoy\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.554268 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.554125    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-credential-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.554408 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.554267    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/602de6ee-5748-44d1-b389-7124d80e521b-istio-podinfo\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.554408 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.554302    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-workload-certs\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.554408 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.554320    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-workload-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.554408 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.554358    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-istio-data\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.554586 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.554508    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqxp5\" (UniqueName: \"kubernetes.io/projected/602de6ee-5748-44d1-b389-7124d80e521b-kube-api-access-tqxp5\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.554586 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.554523    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-credential-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.554694 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.554673    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-workload-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.554794 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.554763    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/602de6ee-5748-44d1-b389-7124d80e521b-istiod-ca-cert\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.556450 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.556429    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-istio-envoy\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.556772 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.556751    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/602de6ee-5748-44d1-b389-7124d80e521b-istio-podinfo\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.611064 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.611025    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/602de6ee-5748-44d1-b389-7124d80e521b-istio-token\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.612340 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.612320    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqxp5\" (UniqueName: \"kubernetes.io/projected/602de6ee-5748-44d1-b389-7124d80e521b-kube-api-access-tqxp5\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:19.904791 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:19.904710    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:20.323147 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:16:20.323118    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602de6ee_5748_44d1_b389_7124d80e521b.slice/crio-951b646c00ecf5463663e9a3d5c38608d50a82db0e50cf799e6c71e10723376f WatchSource:0}: Error finding container 951b646c00ecf5463663e9a3d5c38608d50a82db0e50cf799e6c71e10723376f: Status 404 returned error can't find the container with id 951b646c00ecf5463663e9a3d5c38608d50a82db0e50cf799e6c71e10723376f
Nov 02 23:16:20.323497 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.323396    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"]
Nov 02 23:16:20.325672 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.325641    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:16:20.325758 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.325707    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:16:20.325758 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.325737    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:16:20.448380 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.448347    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:20.503932 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.503899    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk" event={"ID":"602de6ee-5748-44d1-b389-7124d80e521b","Type":"ContainerStarted","Data":"1d2490d1278ee6eb8ab4e430d731848635d5fb526d2d655ea46d6f87c65a6e66"}
Nov 02 23:16:20.504126 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.503952    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk" event={"ID":"602de6ee-5748-44d1-b389-7124d80e521b","Type":"ContainerStarted","Data":"951b646c00ecf5463663e9a3d5c38608d50a82db0e50cf799e6c71e10723376f"}
Nov 02 23:16:20.649013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.648874    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk" podStartSLOduration=1.64885477 podStartE2EDuration="1.64885477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:16:20.644242152 +0000 UTC m=+1420.212596932" watchObservedRunningTime="2026-11-02 23:16:20.64885477 +0000 UTC m=+1420.217209563"
Nov 02 23:16:20.649245 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.649214    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:20.835207 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.835178    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8"]
Nov 02 23:16:20.835952 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.835936    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e08c76d9-473c-4282-b75b-87e625e8c9b5" containerName="cleanup"
Nov 02 23:16:20.838224 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.838210    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8"
Nov 02 23:16:20.891420 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.890465    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htwhl\" (UniqueName: \"kubernetes.io/projected/6a5dd45a-b902-476e-853f-7e09438bc122-kube-api-access-htwhl\") pod \"payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8\" (UID: \"6a5dd45a-b902-476e-853f-7e09438bc122\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8"
Nov 02 23:16:20.891420 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.890742    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5dd45a-b902-476e-853f-7e09438bc122-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8\" (UID: \"6a5dd45a-b902-476e-853f-7e09438bc122\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8"
Nov 02 23:16:20.891740 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.891702    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ede388\""
Nov 02 23:16:20.899492 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.899402    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8"]
Nov 02 23:16:20.900232 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.900206    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ede388-dockercfg-5wzkp\""
Nov 02 23:16:20.905329 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.905301    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:20.991624 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.991590    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5dd45a-b902-476e-853f-7e09438bc122-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8\" (UID: \"6a5dd45a-b902-476e-853f-7e09438bc122\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8"
Nov 02 23:16:20.991824 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.991641    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htwhl\" (UniqueName: \"kubernetes.io/projected/6a5dd45a-b902-476e-853f-7e09438bc122-kube-api-access-htwhl\") pod \"payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8\" (UID: \"6a5dd45a-b902-476e-853f-7e09438bc122\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8"
Nov 02 23:16:20.992378 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:20.992352    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5dd45a-b902-476e-853f-7e09438bc122-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8\" (UID: \"6a5dd45a-b902-476e-853f-7e09438bc122\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8"
Nov 02 23:16:21.167879 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:21.167790    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htwhl\" (UniqueName: \"kubernetes.io/projected/6a5dd45a-b902-476e-853f-7e09438bc122-kube-api-access-htwhl\") pod \"payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8\" (UID: \"6a5dd45a-b902-476e-853f-7e09438bc122\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8"
Nov 02 23:16:21.454010 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:21.453155    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8"
Nov 02 23:16:21.843426 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:16:21.843391    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5dd45a_b902_476e_853f_7e09438bc122.slice/crio-3d303db536063061f237d686a0d9017e474a270255da5733aacff538b850ffae WatchSource:0}: Error finding container 3d303db536063061f237d686a0d9017e474a270255da5733aacff538b850ffae: Status 404 returned error can't find the container with id 3d303db536063061f237d686a0d9017e474a270255da5733aacff538b850ffae
Nov 02 23:16:21.844085 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:21.843956    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8"]
Nov 02 23:16:21.906927 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:21.906888    4294 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 23:16:21.907127 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:21.906954    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk" podUID="602de6ee-5748-44d1-b389-7124d80e521b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 23:16:22.518362 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:22.518323    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8" event={"ID":"6a5dd45a-b902-476e-853f-7e09438bc122","Type":"ContainerStarted","Data":"091e0561ba36d0d4b915872171ae734e70a64d49a109a3c41db8cc347403711b"}
Nov 02 23:16:22.518911 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:22.518395    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8" event={"ID":"6a5dd45a-b902-476e-853f-7e09438bc122","Type":"ContainerStarted","Data":"3d303db536063061f237d686a0d9017e474a270255da5733aacff538b850ffae"}
Nov 02 23:16:22.518911 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:22.518634    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8"
Nov 02 23:16:22.610044 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:22.609949    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:22.635375 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:22.635317    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8" podStartSLOduration=2.6353048980000002 podStartE2EDuration="2.635304898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:16:22.634941341 +0000 UTC m=+1422.203296127" watchObservedRunningTime="2026-11-02 23:16:22.635304898 +0000 UTC m=+1422.203659685"
Nov 02 23:16:22.906044 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:22.905942    4294 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 23:16:22.906044 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:22.906024    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk" podUID="602de6ee-5748-44d1-b389-7124d80e521b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 23:16:23.916412 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:23.916370    4294 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 23:16:23.917008 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:23.916950    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk" podUID="602de6ee-5748-44d1-b389-7124d80e521b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 23:16:23.997947 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:23.997880    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:24.916410 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:24.916376    4294 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 23:16:24.916704 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:24.916671    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk" podUID="602de6ee-5748-44d1-b389-7124d80e521b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 23:16:25.908500 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:25.908461    4294 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 23:16:25.908823 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:25.908774    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk" podUID="602de6ee-5748-44d1-b389-7124d80e521b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 23:16:26.905802 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:26.905762    4294 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 02 23:16:26.906485 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:26.906434    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk" podUID="602de6ee-5748-44d1-b389-7124d80e521b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 02 23:16:28.059879 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:28.059831    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:28.227871 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:28.227823    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:28.228695 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:28.228669    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:28.229644 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:28.229625    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:16:29.314764 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:29.314727    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:31.261933 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:31.261901    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:31.434034 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:31.434002    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:32.982830 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:32.982795    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:33.536800 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:33.536766    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8"
Nov 02 23:16:37.881577 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:37.881537    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:41.752053 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:16:41.752014    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:16:42.063035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:42.062948    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:42.346228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:42.346138    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:42.973400 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:42.973358    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:48.187395 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:48.187347    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:53.113898 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:53.113867    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:54.073700 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:54.073668    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:55.578099 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:55.578065    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8"]
Nov 02 23:16:55.754711 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:55.754678    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:55.950394 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:55.950299    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:56.103952 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:56.103921    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:16:58.427415 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:16:58.427376    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:00.578470 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:00.578416    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8" podUID="c23f0ba4-8743-4fae-bdbe-307266213d9e" containerName="payload-processing" containerID="cri-o://42559665b1365922fc2ffe941d3b5804a0dab2d915f23211a98ff92d352baef9" gracePeriod=25
Nov 02 23:17:01.188653 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:01.188619    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:01.350626 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:01.350592    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:01.797595 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:01.797547    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:03.418790 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:03.418758    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:03.430823 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:03.430792    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8" podUID="c23f0ba4-8743-4fae-bdbe-307266213d9e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:17:03.616675 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:03.616639    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:03.775793 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:03.775748    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:07.783646 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:07.783595    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:09.747317 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:09.747272    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:11.758627 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:17:11.758586    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:17:13.430834 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:13.430799    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8" podUID="c23f0ba4-8743-4fae-bdbe-307266213d9e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:17:13.796026 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:13.795975    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:18.793343 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:18.793307    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:20.847939 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:20.847871    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8"
Nov 02 23:17:20.858344 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:20.858321    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c23f0ba4-8743-4fae-bdbe-307266213d9e-kube-api-access-662dv\" (UniqueName: \"kubernetes.io/projected/c23f0ba4-8743-4fae-bdbe-307266213d9e-kube-api-access-662dv\") pod \"c23f0ba4-8743-4fae-bdbe-307266213d9e\" (UID: \"c23f0ba4-8743-4fae-bdbe-307266213d9e\") "
Nov 02 23:17:20.858488 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:20.858475    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c23f0ba4-8743-4fae-bdbe-307266213d9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23f0ba4-8743-4fae-bdbe-307266213d9e-plugins-config-volume\") pod \"c23f0ba4-8743-4fae-bdbe-307266213d9e\" (UID: \"c23f0ba4-8743-4fae-bdbe-307266213d9e\") "
Nov 02 23:17:20.858809 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:20.858787    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c23f0ba4-8743-4fae-bdbe-307266213d9e-plugins-config-volume" pod "c23f0ba4-8743-4fae-bdbe-307266213d9e" (UID: "c23f0ba4-8743-4fae-bdbe-307266213d9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:17:20.860597 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:20.860573    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c23f0ba4-8743-4fae-bdbe-307266213d9e-kube-api-access-662dv" pod "c23f0ba4-8743-4fae-bdbe-307266213d9e" (UID: "c23f0ba4-8743-4fae-bdbe-307266213d9e"). InnerVolumeSpecName "kube-api-access-662dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:17:20.960722 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:20.960692    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-662dv\" (UniqueName: \"kubernetes.io/projected/c23f0ba4-8743-4fae-bdbe-307266213d9e-kube-api-access-662dv\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:17:20.960722 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:20.960722    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23f0ba4-8743-4fae-bdbe-307266213d9e-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:17:20.999752 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:20.999715    4294 generic.go:356] "Generic (PLEG): container finished" podID="c23f0ba4-8743-4fae-bdbe-307266213d9e" containerID="42559665b1365922fc2ffe941d3b5804a0dab2d915f23211a98ff92d352baef9" exitCode=0
Nov 02 23:17:20.999940 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:20.999782    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8"
Nov 02 23:17:20.999940 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:20.999860    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8" event={"ID":"c23f0ba4-8743-4fae-bdbe-307266213d9e","Type":"ContainerDied","Data":"42559665b1365922fc2ffe941d3b5804a0dab2d915f23211a98ff92d352baef9"}
Nov 02 23:17:20.999940 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:20.999905    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8" event={"ID":"c23f0ba4-8743-4fae-bdbe-307266213d9e","Type":"ContainerDied","Data":"920a056f5ba9b8c20376589a2c5ae05a8fafa7e8c5f2a83b1de689449699c78a"}
Nov 02 23:17:20.999940 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:20.999935    4294 scope.go:122] "RemoveContainer" containerID="42559665b1365922fc2ffe941d3b5804a0dab2d915f23211a98ff92d352baef9"
Nov 02 23:17:21.015449 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:21.015420    4294 scope.go:122] "RemoveContainer" containerID="42559665b1365922fc2ffe941d3b5804a0dab2d915f23211a98ff92d352baef9"
Nov 02 23:17:21.015797 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:17:21.015756    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42559665b1365922fc2ffe941d3b5804a0dab2d915f23211a98ff92d352baef9\": container with ID starting with 42559665b1365922fc2ffe941d3b5804a0dab2d915f23211a98ff92d352baef9 not found: ID does not exist" containerID="42559665b1365922fc2ffe941d3b5804a0dab2d915f23211a98ff92d352baef9"
Nov 02 23:17:21.015900 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:21.015800    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42559665b1365922fc2ffe941d3b5804a0dab2d915f23211a98ff92d352baef9"} err="failed to get container status \"42559665b1365922fc2ffe941d3b5804a0dab2d915f23211a98ff92d352baef9\": rpc error: code = NotFound desc = could not find container \"42559665b1365922fc2ffe941d3b5804a0dab2d915f23211a98ff92d352baef9\": container with ID starting with 42559665b1365922fc2ffe941d3b5804a0dab2d915f23211a98ff92d352baef9 not found: ID does not exist"
Nov 02 23:17:21.161252 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:21.161171    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8"]
Nov 02 23:17:21.164437 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:21.164413    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kd2r8"]
Nov 02 23:17:22.549956 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:22.549915    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c23f0ba4-8743-4fae-bdbe-307266213d9e" path="/var/lib/kubelet/pods/c23f0ba4-8743-4fae-bdbe-307266213d9e/volumes"
Nov 02 23:17:23.052674 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:23.052633    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:23.287082 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:23.287039    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:24.903127 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:24.903054    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:28.637036 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:28.637004    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:29.350421 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:29.350385    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:29.767291 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:29.767260    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:29.947068 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:29.945161    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:30.115287 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:30.115212    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:30.461511 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:30.461431    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:30.628362 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:30.628331    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:32.086551 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:32.086516    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:32.661753 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:32.661721    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:33.337261 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:33.337228    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:34.031493 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:34.031461    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:34.232541 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:34.232505    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:34.531768 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:34.531736    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:35.115390 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:35.115345    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:37.174952 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:37.174921    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:37.346794 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:37.346759    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:37.561263 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:37.561233    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2"]
Nov 02 23:17:37.561859 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:37.561845    4294 cpu_manager.go:414] "containerMap: removing container" podUID="c23f0ba4-8743-4fae-bdbe-307266213d9e" containerName="payload-processing"
Nov 02 23:17:37.561910 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:37.561859    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c23f0ba4-8743-4fae-bdbe-307266213d9e" containerName="payload-processing"
Nov 02 23:17:37.561961 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:37.561952    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c23f0ba4-8743-4fae-bdbe-307266213d9e" containerName="payload-processing"
Nov 02 23:17:37.566804 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:37.566776    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2"
Nov 02 23:17:37.636203 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:37.636163    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c12aac19-4212-4954-bc46-10688e21eb09-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2\" (UID: \"c12aac19-4212-4954-bc46-10688e21eb09\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2"
Nov 02 23:17:37.636607 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:37.636581    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct4g6\" (UniqueName: \"kubernetes.io/projected/c12aac19-4212-4954-bc46-10688e21eb09-kube-api-access-ct4g6\") pod \"payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2\" (UID: \"c12aac19-4212-4954-bc46-10688e21eb09\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2"
Nov 02 23:17:37.655923 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:37.655880    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2"]
Nov 02 23:17:37.660888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:37.660863    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-68e067\""
Nov 02 23:17:37.661102 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:37.660863    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-68e067-dockercfg-qknh8\""
Nov 02 23:17:37.738462 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:37.738410    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct4g6\" (UniqueName: \"kubernetes.io/projected/c12aac19-4212-4954-bc46-10688e21eb09-kube-api-access-ct4g6\") pod \"payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2\" (UID: \"c12aac19-4212-4954-bc46-10688e21eb09\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2"
Nov 02 23:17:37.738670 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:37.738532    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c12aac19-4212-4954-bc46-10688e21eb09-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2\" (UID: \"c12aac19-4212-4954-bc46-10688e21eb09\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2"
Nov 02 23:17:37.739610 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:37.739582    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c12aac19-4212-4954-bc46-10688e21eb09-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2\" (UID: \"c12aac19-4212-4954-bc46-10688e21eb09\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2"
Nov 02 23:17:37.793725 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:37.793688    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct4g6\" (UniqueName: \"kubernetes.io/projected/c12aac19-4212-4954-bc46-10688e21eb09-kube-api-access-ct4g6\") pod \"payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2\" (UID: \"c12aac19-4212-4954-bc46-10688e21eb09\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2"
Nov 02 23:17:37.882585 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:37.882498    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2"
Nov 02 23:17:38.272375 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:38.272347    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2"]
Nov 02 23:17:38.272813 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:17:38.272533    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12aac19_4212_4954_bc46_10688e21eb09.slice/crio-158f75f8ebe87042d37a064c5f2ea892656cafbe705a04c2a3b893d21da946ca WatchSource:0}: Error finding container 158f75f8ebe87042d37a064c5f2ea892656cafbe705a04c2a3b893d21da946ca: Status 404 returned error can't find the container with id 158f75f8ebe87042d37a064c5f2ea892656cafbe705a04c2a3b893d21da946ca
Nov 02 23:17:39.121515 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:39.121479    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2" event={"ID":"c12aac19-4212-4954-bc46-10688e21eb09","Type":"ContainerStarted","Data":"402eee30ee17c4bb7ab0d20b941fc86de4b5f9c86ee7c8d0b243952836998063"}
Nov 02 23:17:39.121708 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:39.121539    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2" event={"ID":"c12aac19-4212-4954-bc46-10688e21eb09","Type":"ContainerStarted","Data":"158f75f8ebe87042d37a064c5f2ea892656cafbe705a04c2a3b893d21da946ca"}
Nov 02 23:17:39.121708 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:39.121579    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2"
Nov 02 23:17:39.205715 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:39.205681    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:39.230737 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:39.230688    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2" podStartSLOduration=2.230673311 podStartE2EDuration="2.230673311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:17:39.229242955 +0000 UTC m=+1498.797597742" watchObservedRunningTime="2026-11-02 23:17:39.230673311 +0000 UTC m=+1498.799028098"
Nov 02 23:17:40.570855 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:40.570795    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:40.785678 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:40.785646    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:40.987348 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:40.987311    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:41.754512 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:17:41.754470    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:17:45.154953 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:45.154917    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:46.027395 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:46.027353    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:50.130918 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:50.130890    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2"
Nov 02 23:17:51.426478 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:51.426446    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:53.858214 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:53.858173    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:54.556116 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:54.556081    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:55.169961 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:55.169909    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:55.532654 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:55.532614    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:56.826338 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:56.826295    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:59.126155 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:59.126106    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:59.305781 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:59.305747    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:17:59.596888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:17:59.596854    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:02.081662 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:02.081631    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:02.360606 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:02.360513    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:02.523433 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:02.523399    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:06.782328 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:06.782296    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:07.090562 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:07.090405    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:08.542440 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:08.542407    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:08.699265 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:08.699225    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:08.861672 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:08.861584    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:11.752109 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:18:11.752073    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:18:12.870313 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:12.870283    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:13.357449 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:13.357417    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:15.216698 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:15.216658    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:15.383874 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:15.383841    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:15.545665 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:15.545622    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:17.181970 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:17.181936    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:17.361029 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:17.360969    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:17.781658 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:17.781628    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:17.983114 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:17.983079    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:19.414140 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:19.414107    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:19.583814 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:19.583781    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:19.790002 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:19.789499    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:19.889930 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:19.889896    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt"]
Nov 02 23:18:19.894933 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:19.894879    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt"
Nov 02 23:18:19.945931 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:19.945898    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt"]
Nov 02 23:18:19.970561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:19.970520    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:19.990033 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:19.989141    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e953257b\""
Nov 02 23:18:19.990033 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:19.989187    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e953257b-dockercfg-tz4l5\""
Nov 02 23:18:20.059608 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:20.059522    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdh5c\" (UniqueName: \"kubernetes.io/projected/d562d136-00fe-4689-8072-05b919e15697-kube-api-access-vdh5c\") pod \"payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt\" (UID: \"d562d136-00fe-4689-8072-05b919e15697\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt"
Nov 02 23:18:20.059785 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:20.059747    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d562d136-00fe-4689-8072-05b919e15697-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt\" (UID: \"d562d136-00fe-4689-8072-05b919e15697\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt"
Nov 02 23:18:20.060737 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:20.060708    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq"]
Nov 02 23:18:20.065782 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:20.065757    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq"
Nov 02 23:18:20.116050 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:20.115571    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq"]
Nov 02 23:18:20.166003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:20.164015    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rth9z\" (UniqueName: \"kubernetes.io/projected/20c964fe-e6c9-4a32-bea1-e9e29845a78e-kube-api-access-rth9z\") pod \"payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq\" (UID: \"20c964fe-e6c9-4a32-bea1-e9e29845a78e\") " pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq"
Nov 02 23:18:20.166003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:20.164212    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c964fe-e6c9-4a32-bea1-e9e29845a78e-plugins-config-volume\") pod \"payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq\" (UID: \"20c964fe-e6c9-4a32-bea1-e9e29845a78e\") " pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq"
Nov 02 23:18:20.166003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:20.164256    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d562d136-00fe-4689-8072-05b919e15697-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt\" (UID: \"d562d136-00fe-4689-8072-05b919e15697\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt"
Nov 02 23:18:20.166003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:20.164486    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdh5c\" (UniqueName: \"kubernetes.io/projected/d562d136-00fe-4689-8072-05b919e15697-kube-api-access-vdh5c\") pod \"payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt\" (UID: \"d562d136-00fe-4689-8072-05b919e15697\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt"
Nov 02 23:18:20.166003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:20.165907    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d562d136-00fe-4689-8072-05b919e15697-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt\" (UID: \"d562d136-00fe-4689-8072-05b919e15697\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt"
Nov 02 23:18:20.220631 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:20.220597    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdh5c\" (UniqueName: \"kubernetes.io/projected/d562d136-00fe-4689-8072-05b919e15697-kube-api-access-vdh5c\") pod \"payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt\" (UID: \"d562d136-00fe-4689-8072-05b919e15697\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt"
Nov 02 23:18:20.265846 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:20.265810    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rth9z\" (UniqueName: \"kubernetes.io/projected/20c964fe-e6c9-4a32-bea1-e9e29845a78e-kube-api-access-rth9z\") pod \"payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq\" (UID: \"20c964fe-e6c9-4a32-bea1-e9e29845a78e\") " pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq"
Nov 02 23:18:20.266035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:20.265967    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c964fe-e6c9-4a32-bea1-e9e29845a78e-plugins-config-volume\") pod \"payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq\" (UID: \"20c964fe-e6c9-4a32-bea1-e9e29845a78e\") " pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq"
Nov 02 23:18:20.267467 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:20.267438    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c964fe-e6c9-4a32-bea1-e9e29845a78e-plugins-config-volume\") pod \"payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq\" (UID: \"20c964fe-e6c9-4a32-bea1-e9e29845a78e\") " pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq"
Nov 02 23:18:20.320894 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:20.320824    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rth9z\" (UniqueName: \"kubernetes.io/projected/20c964fe-e6c9-4a32-bea1-e9e29845a78e-kube-api-access-rth9z\") pod \"payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq\" (UID: \"20c964fe-e6c9-4a32-bea1-e9e29845a78e\") " pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq"
Nov 02 23:18:20.383113 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:20.383065    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq"
Nov 02 23:18:20.511437 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:20.510961    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt"
Nov 02 23:18:20.728431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:20.728387    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:20.778603 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:20.778574    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq"]
Nov 02 23:18:20.780544 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:18:20.780507    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c964fe_e6c9_4a32_bea1_e9e29845a78e.slice/crio-611549435cdd72fc18b7c1f7b3e89b3f588baf76aa8289c4ab534f9d37b7eaec WatchSource:0}: Error finding container 611549435cdd72fc18b7c1f7b3e89b3f588baf76aa8289c4ab534f9d37b7eaec: Status 404 returned error can't find the container with id 611549435cdd72fc18b7c1f7b3e89b3f588baf76aa8289c4ab534f9d37b7eaec
Nov 02 23:18:20.901885 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:20.901857    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt"]
Nov 02 23:18:20.902635 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:18:20.902611    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd562d136_00fe_4689_8072_05b919e15697.slice/crio-857fed60902e477ad26b029321d0744b5716b188ba2d9cae27c48271f3334edc WatchSource:0}: Error finding container 857fed60902e477ad26b029321d0744b5716b188ba2d9cae27c48271f3334edc: Status 404 returned error can't find the container with id 857fed60902e477ad26b029321d0744b5716b188ba2d9cae27c48271f3334edc
Nov 02 23:18:21.438353 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:21.438318    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq" event={"ID":"20c964fe-e6c9-4a32-bea1-e9e29845a78e","Type":"ContainerStarted","Data":"570ec042b3631c6ce8206dfcc3f4c769922c159be3f688c66fbbb96ef2ea9ad2"}
Nov 02 23:18:21.438550 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:21.438375    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq" event={"ID":"20c964fe-e6c9-4a32-bea1-e9e29845a78e","Type":"ContainerStarted","Data":"611549435cdd72fc18b7c1f7b3e89b3f588baf76aa8289c4ab534f9d37b7eaec"}
Nov 02 23:18:21.438550 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:21.438532    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq"
Nov 02 23:18:21.440159 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:21.440133    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt" event={"ID":"d562d136-00fe-4689-8072-05b919e15697","Type":"ContainerStarted","Data":"3179f7f05d9352b52d634663d70d7e909fa456433bfeb3192d032b93508c0b37"}
Nov 02 23:18:21.440279 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:21.440171    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt" event={"ID":"d562d136-00fe-4689-8072-05b919e15697","Type":"ContainerStarted","Data":"857fed60902e477ad26b029321d0744b5716b188ba2d9cae27c48271f3334edc"}
Nov 02 23:18:21.440279 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:21.440226    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt"
Nov 02 23:18:21.551372 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:21.551313    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq" podStartSLOduration=1.551299079 podStartE2EDuration="1.551299079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:18:21.5491799 +0000 UTC m=+1541.117534689" watchObservedRunningTime="2026-11-02 23:18:21.551299079 +0000 UTC m=+1541.119653890"
Nov 02 23:18:21.658627 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:21.658565    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt" podStartSLOduration=2.658545923 podStartE2EDuration="2.658545923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:18:21.658017029 +0000 UTC m=+1541.226371817" watchObservedRunningTime="2026-11-02 23:18:21.658545923 +0000 UTC m=+1541.226900711"
Nov 02 23:18:25.807521 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:25.807473    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:30.121972 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:30.121926    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:30.280025 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:30.279991    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:30.283087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:30.283064    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"]
Nov 02 23:18:30.283299 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:30.283276    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96" podUID="09f13590-7f6c-4c1c-90c2-b59751b42720" containerName="istio-proxy" containerID="cri-o://2d35dec0330a9e301a327b187b2f15a71161acc379b0df672ee1c591d40b3b2c" gracePeriod=30
Nov 02 23:18:30.687943 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:30.687889    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:32.449844 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:32.449814    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq"
Nov 02 23:18:32.450419 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:32.449882    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt"
Nov 02 23:18:35.534293 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.534259    4294 generic.go:356] "Generic (PLEG): container finished" podID="09f13590-7f6c-4c1c-90c2-b59751b42720" containerID="2d35dec0330a9e301a327b187b2f15a71161acc379b0df672ee1c591d40b3b2c" exitCode=0
Nov 02 23:18:35.534618 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.534323    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96" event={"ID":"09f13590-7f6c-4c1c-90c2-b59751b42720","Type":"ContainerDied","Data":"2d35dec0330a9e301a327b187b2f15a71161acc379b0df672ee1c591d40b3b2c"}
Nov 02 23:18:35.534618 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.534375    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96" event={"ID":"09f13590-7f6c-4c1c-90c2-b59751b42720","Type":"ContainerDied","Data":"220b27fab968abab7505fb48e69af0bd7254d8702ff0c74b8b3413c722f004e3"}
Nov 02 23:18:35.534618 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.534391    4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="220b27fab968abab7505fb48e69af0bd7254d8702ff0c74b8b3413c722f004e3"
Nov 02 23:18:35.552876 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.552853    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:18:35.576142 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.576110    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:35.703435 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.703345    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f13590-7f6c-4c1c-90c2-b59751b42720-kube-api-access-rvt46\" (UniqueName: \"kubernetes.io/projected/09f13590-7f6c-4c1c-90c2-b59751b42720-kube-api-access-rvt46\") pod \"09f13590-7f6c-4c1c-90c2-b59751b42720\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") "
Nov 02 23:18:35.703602 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.703487    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-workload-socket\") pod \"09f13590-7f6c-4c1c-90c2-b59751b42720\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") "
Nov 02 23:18:35.703602 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.703535    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-podinfo\") pod \"09f13590-7f6c-4c1c-90c2-b59751b42720\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") "
Nov 02 23:18:35.703602 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.703574    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-envoy\") pod \"09f13590-7f6c-4c1c-90c2-b59751b42720\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") "
Nov 02 23:18:35.703602 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.703603    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-workload-certs\") pod \"09f13590-7f6c-4c1c-90c2-b59751b42720\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") "
Nov 02 23:18:35.703825 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.703631    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-token\" (UniqueName: \"kubernetes.io/projected/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-token\") pod \"09f13590-7f6c-4c1c-90c2-b59751b42720\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") "
Nov 02 23:18:35.703825 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.703679    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-credential-socket\") pod \"09f13590-7f6c-4c1c-90c2-b59751b42720\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") "
Nov 02 23:18:35.703825 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.703783    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-workload-socket" pod "09f13590-7f6c-4c1c-90c2-b59751b42720" (UID: "09f13590-7f6c-4c1c-90c2-b59751b42720"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:18:35.703971 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.703847    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-data\") pod \"09f13590-7f6c-4c1c-90c2-b59751b42720\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") "
Nov 02 23:18:35.703971 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.703889    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-workload-certs" pod "09f13590-7f6c-4c1c-90c2-b59751b42720" (UID: "09f13590-7f6c-4c1c-90c2-b59751b42720"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:18:35.703971 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.703915    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-credential-socket" pod "09f13590-7f6c-4c1c-90c2-b59751b42720" (UID: "09f13590-7f6c-4c1c-90c2-b59751b42720"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:18:35.704143 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.703971    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f13590-7f6c-4c1c-90c2-b59751b42720-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f13590-7f6c-4c1c-90c2-b59751b42720-istiod-ca-cert\") pod \"09f13590-7f6c-4c1c-90c2-b59751b42720\" (UID: \"09f13590-7f6c-4c1c-90c2-b59751b42720\") "
Nov 02 23:18:35.704190 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.704161    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-data" pod "09f13590-7f6c-4c1c-90c2-b59751b42720" (UID: "09f13590-7f6c-4c1c-90c2-b59751b42720"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:18:35.704414 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.704391    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f13590-7f6c-4c1c-90c2-b59751b42720-istiod-ca-cert" pod "09f13590-7f6c-4c1c-90c2-b59751b42720" (UID: "09f13590-7f6c-4c1c-90c2-b59751b42720"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:18:35.704627 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.704609    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:18:35.704701 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.704633    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:18:35.704701 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.704648    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:18:35.704701 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.704662    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:18:35.704701 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.704676    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f13590-7f6c-4c1c-90c2-b59751b42720-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:18:35.706105 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.706075    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-envoy" pod "09f13590-7f6c-4c1c-90c2-b59751b42720" (UID: "09f13590-7f6c-4c1c-90c2-b59751b42720"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:18:35.706215 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.706110    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f13590-7f6c-4c1c-90c2-b59751b42720-kube-api-access-rvt46" pod "09f13590-7f6c-4c1c-90c2-b59751b42720" (UID: "09f13590-7f6c-4c1c-90c2-b59751b42720"). InnerVolumeSpecName "kube-api-access-rvt46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:18:35.706263 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.706253    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-token" pod "09f13590-7f6c-4c1c-90c2-b59751b42720" (UID: "09f13590-7f6c-4c1c-90c2-b59751b42720"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:18:35.706301 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.706254    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-podinfo" pod "09f13590-7f6c-4c1c-90c2-b59751b42720" (UID: "09f13590-7f6c-4c1c-90c2-b59751b42720"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:18:35.806384 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.806351    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:18:35.806384 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.806381    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:18:35.806384 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.806391    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f13590-7f6c-4c1c-90c2-b59751b42720-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:18:35.806611 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:35.806400    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvt46\" (UniqueName: \"kubernetes.io/projected/09f13590-7f6c-4c1c-90c2-b59751b42720-kube-api-access-rvt46\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:18:36.540891 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:36.540857    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"
Nov 02 23:18:36.746478 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:36.746445    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"]
Nov 02 23:18:36.793479 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:36.793394    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-x7g96"]
Nov 02 23:18:38.548897 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:38.548865    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f13590-7f6c-4c1c-90c2-b59751b42720" path="/var/lib/kubelet/pods/09f13590-7f6c-4c1c-90c2-b59751b42720/volumes"
Nov 02 23:18:40.485219 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:40.485179    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:40.841657 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:40.841622    4294 scope.go:122] "RemoveContainer" containerID="2d35dec0330a9e301a327b187b2f15a71161acc379b0df672ee1c591d40b3b2c"
Nov 02 23:18:41.750756 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:18:41.750724    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:18:45.322818 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:45.322777    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:45.492872 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:45.492835    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:45.654240 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:45.654159    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:50.698057 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:50.698002    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:54.494340 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:54.494252    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:55.130699 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:55.130548    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:55.702349 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:55.702320    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:55.858026 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:55.857949    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:56.026948 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:56.026906    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:59.406049 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:59.406011    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:59.602939 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:59.602904    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:59.777455 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:59.777421    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:18:59.941052 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:59.941005    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"]
Nov 02 23:18:59.941321 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:18:59.941285    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w" podUID="50895b59-7759-4a46-aa8f-a1d691c255de" containerName="istio-proxy" containerID="cri-o://e0b8cb8ebad878d355e5d3cdcd1bc4cc6acd8333220d5807dab36e2e89790bed" gracePeriod=30
Nov 02 23:19:00.906093 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:00.906055    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:04.206289 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:04.206257    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:04.381903 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:04.381866    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:05.211868 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.211806    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:19:05.368598 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.368564    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-workload-socket\") pod \"50895b59-7759-4a46-aa8f-a1d691c255de\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") "
Nov 02 23:19:05.368763 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.368667    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50895b59-7759-4a46-aa8f-a1d691c255de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50895b59-7759-4a46-aa8f-a1d691c255de-istiod-ca-cert\") pod \"50895b59-7759-4a46-aa8f-a1d691c255de\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") "
Nov 02 23:19:05.368763 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.368734    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-istio-envoy\") pod \"50895b59-7759-4a46-aa8f-a1d691c255de\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") "
Nov 02 23:19:05.368763 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.368760    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50895b59-7759-4a46-aa8f-a1d691c255de-kube-api-access-d5vhs\" (UniqueName: \"kubernetes.io/projected/50895b59-7759-4a46-aa8f-a1d691c255de-kube-api-access-d5vhs\") pod \"50895b59-7759-4a46-aa8f-a1d691c255de\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") "
Nov 02 23:19:05.368902 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.368781    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50895b59-7759-4a46-aa8f-a1d691c255de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50895b59-7759-4a46-aa8f-a1d691c255de-istio-podinfo\") pod \"50895b59-7759-4a46-aa8f-a1d691c255de\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") "
Nov 02 23:19:05.368902 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.368811    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-workload-certs\") pod \"50895b59-7759-4a46-aa8f-a1d691c255de\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") "
Nov 02 23:19:05.369035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.368904    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-istio-data\") pod \"50895b59-7759-4a46-aa8f-a1d691c255de\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") "
Nov 02 23:19:05.369035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.368882    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-workload-socket" pod "50895b59-7759-4a46-aa8f-a1d691c255de" (UID: "50895b59-7759-4a46-aa8f-a1d691c255de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:05.369035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.369015    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50895b59-7759-4a46-aa8f-a1d691c255de-istio-token\" (UniqueName: \"kubernetes.io/projected/50895b59-7759-4a46-aa8f-a1d691c255de-istio-token\") pod \"50895b59-7759-4a46-aa8f-a1d691c255de\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") "
Nov 02 23:19:05.369201 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.369083    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-credential-socket\") pod \"50895b59-7759-4a46-aa8f-a1d691c255de\" (UID: \"50895b59-7759-4a46-aa8f-a1d691c255de\") "
Nov 02 23:19:05.369201 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.369096    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50895b59-7759-4a46-aa8f-a1d691c255de-istiod-ca-cert" pod "50895b59-7759-4a46-aa8f-a1d691c255de" (UID: "50895b59-7759-4a46-aa8f-a1d691c255de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:19:05.369299 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.369226    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-istio-data" pod "50895b59-7759-4a46-aa8f-a1d691c255de" (UID: "50895b59-7759-4a46-aa8f-a1d691c255de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:05.369451 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.369408    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-credential-socket" pod "50895b59-7759-4a46-aa8f-a1d691c255de" (UID: "50895b59-7759-4a46-aa8f-a1d691c255de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:05.369567 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.369468    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-workload-certs" pod "50895b59-7759-4a46-aa8f-a1d691c255de" (UID: "50895b59-7759-4a46-aa8f-a1d691c255de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:05.369918 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.369897    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:05.369973 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.369927    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50895b59-7759-4a46-aa8f-a1d691c255de-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:05.369973 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.369943    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:05.369973 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.369958    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:05.369973 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.369974    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:05.371184 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.371161    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-istio-envoy" pod "50895b59-7759-4a46-aa8f-a1d691c255de" (UID: "50895b59-7759-4a46-aa8f-a1d691c255de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:05.371276 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.371228    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50895b59-7759-4a46-aa8f-a1d691c255de-istio-token" pod "50895b59-7759-4a46-aa8f-a1d691c255de" (UID: "50895b59-7759-4a46-aa8f-a1d691c255de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:19:05.371598 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.371578    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50895b59-7759-4a46-aa8f-a1d691c255de-istio-podinfo" pod "50895b59-7759-4a46-aa8f-a1d691c255de" (UID: "50895b59-7759-4a46-aa8f-a1d691c255de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:19:05.371668 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.371601    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50895b59-7759-4a46-aa8f-a1d691c255de-kube-api-access-d5vhs" pod "50895b59-7759-4a46-aa8f-a1d691c255de" (UID: "50895b59-7759-4a46-aa8f-a1d691c255de"). InnerVolumeSpecName "kube-api-access-d5vhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:19:05.471888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.471806    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50895b59-7759-4a46-aa8f-a1d691c255de-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:05.471888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.471837    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5vhs\" (UniqueName: \"kubernetes.io/projected/50895b59-7759-4a46-aa8f-a1d691c255de-kube-api-access-d5vhs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:05.471888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.471861    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50895b59-7759-4a46-aa8f-a1d691c255de-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:05.471888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.471870    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50895b59-7759-4a46-aa8f-a1d691c255de-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:05.759198 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.759161    4294 generic.go:356] "Generic (PLEG): container finished" podID="50895b59-7759-4a46-aa8f-a1d691c255de" containerID="e0b8cb8ebad878d355e5d3cdcd1bc4cc6acd8333220d5807dab36e2e89790bed" exitCode=0
Nov 02 23:19:05.759198 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.759190    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w" event={"ID":"50895b59-7759-4a46-aa8f-a1d691c255de","Type":"ContainerDied","Data":"e0b8cb8ebad878d355e5d3cdcd1bc4cc6acd8333220d5807dab36e2e89790bed"}
Nov 02 23:19:05.759448 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.759208    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"
Nov 02 23:19:05.759448 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.759255    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w" event={"ID":"50895b59-7759-4a46-aa8f-a1d691c255de","Type":"ContainerDied","Data":"d625dc49df74a9186dbe1c67fbda7081183b76bface47f1f9549b047f2d5a435"}
Nov 02 23:19:05.759448 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.759279    4294 scope.go:122] "RemoveContainer" containerID="e0b8cb8ebad878d355e5d3cdcd1bc4cc6acd8333220d5807dab36e2e89790bed"
Nov 02 23:19:05.773971 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.773943    4294 scope.go:122] "RemoveContainer" containerID="e0b8cb8ebad878d355e5d3cdcd1bc4cc6acd8333220d5807dab36e2e89790bed"
Nov 02 23:19:05.774595 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:19:05.774563    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0b8cb8ebad878d355e5d3cdcd1bc4cc6acd8333220d5807dab36e2e89790bed\": container with ID starting with e0b8cb8ebad878d355e5d3cdcd1bc4cc6acd8333220d5807dab36e2e89790bed not found: ID does not exist" containerID="e0b8cb8ebad878d355e5d3cdcd1bc4cc6acd8333220d5807dab36e2e89790bed"
Nov 02 23:19:05.774707 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.774608    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0b8cb8ebad878d355e5d3cdcd1bc4cc6acd8333220d5807dab36e2e89790bed"} err="failed to get container status \"e0b8cb8ebad878d355e5d3cdcd1bc4cc6acd8333220d5807dab36e2e89790bed\": rpc error: code = NotFound desc = could not find container \"e0b8cb8ebad878d355e5d3cdcd1bc4cc6acd8333220d5807dab36e2e89790bed\": container with ID starting with e0b8cb8ebad878d355e5d3cdcd1bc4cc6acd8333220d5807dab36e2e89790bed not found: ID does not exist"
Nov 02 23:19:05.917861 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.917824    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"]
Nov 02 23:19:05.965622 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:05.965590    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-6776w"]
Nov 02 23:19:06.082525 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:06.082441    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:06.549313 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:06.549278    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50895b59-7759-4a46-aa8f-a1d691c255de" path="/var/lib/kubelet/pods/50895b59-7759-4a46-aa8f-a1d691c255de/volumes"
Nov 02 23:19:11.080690 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:11.080653    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:11.754622 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:19:11.754569    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:19:15.673178 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:15.673143    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:16.349376 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:16.349341    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:16.537231 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:16.537202    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:16.901593 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:16.901559    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"]
Nov 02 23:19:16.902017 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:16.901818    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff" podUID="51c49a39-5522-4cec-9246-7de8e59d4b9f" containerName="istio-proxy" containerID="cri-o://a5a52c2bb72e102fb1330f9a487b8ebdc6688bed91fc90bfff88230064d29b0c" gracePeriod=30
Nov 02 23:19:16.912323 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:16.912300    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:17.104026 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:17.090445    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:18.693753 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:18.693719    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"]
Nov 02 23:19:18.694265 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:18.693970    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq" podUID="52a13074-c991-4d08-9bfd-c132ffab4c90" containerName="istio-proxy" containerID="cri-o://616c8c98911f49a0700c674bd8f5d70b55d3e18d508732769f818a750011cc0d" gracePeriod=30
Nov 02 23:19:18.742093 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:18.742061    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"]
Nov 02 23:19:18.742576 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:18.742544    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh" podUID="5d17d552-d441-4ed5-a5c1-5a4141808868" containerName="istio-proxy" containerID="cri-o://e89c673aef0fa32412334bcf32446db1b36890b9240345263467d31d72fa3f7b" gracePeriod=30
Nov 02 23:19:18.747791 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:18.747757    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"]
Nov 02 23:19:18.748283 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:18.748253    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp" podUID="9d525079-fdb8-4c0e-bb38-2567fcf4a49a" containerName="istio-proxy" containerID="cri-o://fbd56b7432201d2d341e246841a0cccaa25826338200dcdaabf3688945d3b21b" gracePeriod=30
Nov 02 23:19:18.790402 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:18.790365    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"]
Nov 02 23:19:18.790676 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:18.790651    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp" podUID="67334c5b-8d17-478e-8248-0762c0e1653c" containerName="istio-proxy" containerID="cri-o://0824e99f6eb8e9ee9e5ae119af94a660358274fb1d6f8228ed5d4ea2707d2556" gracePeriod=30
Nov 02 23:19:20.749835 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:20.749801    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:20.948177 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:20.948137    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:21.133093 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:21.132964    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:21.142088 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:21.142049    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 23:19:21.142254 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:21.142126    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp" podUID="9d525079-fdb8-4c0e-bb38-2567fcf4a49a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 23:19:21.676044 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:21.675969    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 23:19:21.676234 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:21.676090    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff" podUID="51c49a39-5522-4cec-9246-7de8e59d4b9f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 23:19:21.738077 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:21.738042    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:22.364535 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.364499    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:19:22.463728 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.463676    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-data\") pod \"51c49a39-5522-4cec-9246-7de8e59d4b9f\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") "
Nov 02 23:19:22.464073 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.464000    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-credential-socket\") pod \"51c49a39-5522-4cec-9246-7de8e59d4b9f\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") "
Nov 02 23:19:22.464372 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.464354    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-workload-socket\") pod \"51c49a39-5522-4cec-9246-7de8e59d4b9f\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") "
Nov 02 23:19:22.464659 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.464645    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-podinfo\") pod \"51c49a39-5522-4cec-9246-7de8e59d4b9f\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") "
Nov 02 23:19:22.464801 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.464786    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-token\" (UniqueName: \"kubernetes.io/projected/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-token\") pod \"51c49a39-5522-4cec-9246-7de8e59d4b9f\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") "
Nov 02 23:19:22.465025 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.465008    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51c49a39-5522-4cec-9246-7de8e59d4b9f-kube-api-access-9d48t\" (UniqueName: \"kubernetes.io/projected/51c49a39-5522-4cec-9246-7de8e59d4b9f-kube-api-access-9d48t\") pod \"51c49a39-5522-4cec-9246-7de8e59d4b9f\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") "
Nov 02 23:19:22.465182 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.465167    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-envoy\") pod \"51c49a39-5522-4cec-9246-7de8e59d4b9f\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") "
Nov 02 23:19:22.465313 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.465299    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51c49a39-5522-4cec-9246-7de8e59d4b9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51c49a39-5522-4cec-9246-7de8e59d4b9f-istiod-ca-cert\") pod \"51c49a39-5522-4cec-9246-7de8e59d4b9f\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") "
Nov 02 23:19:22.465455 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.465440    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-workload-certs\") pod \"51c49a39-5522-4cec-9246-7de8e59d4b9f\" (UID: \"51c49a39-5522-4cec-9246-7de8e59d4b9f\") "
Nov 02 23:19:22.466698 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.466676    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-workload-certs" pod "51c49a39-5522-4cec-9246-7de8e59d4b9f" (UID: "51c49a39-5522-4cec-9246-7de8e59d4b9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:22.470461 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.470110    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-credential-socket" pod "51c49a39-5522-4cec-9246-7de8e59d4b9f" (UID: "51c49a39-5522-4cec-9246-7de8e59d4b9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:22.470461 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.470372    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51c49a39-5522-4cec-9246-7de8e59d4b9f-istiod-ca-cert" pod "51c49a39-5522-4cec-9246-7de8e59d4b9f" (UID: "51c49a39-5522-4cec-9246-7de8e59d4b9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:19:22.470461 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.470433    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-workload-socket" pod "51c49a39-5522-4cec-9246-7de8e59d4b9f" (UID: "51c49a39-5522-4cec-9246-7de8e59d4b9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:22.476197 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.476161    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-data" pod "51c49a39-5522-4cec-9246-7de8e59d4b9f" (UID: "51c49a39-5522-4cec-9246-7de8e59d4b9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:22.499271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.499151    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-envoy" pod "51c49a39-5522-4cec-9246-7de8e59d4b9f" (UID: "51c49a39-5522-4cec-9246-7de8e59d4b9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:22.499271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.499168    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c49a39-5522-4cec-9246-7de8e59d4b9f-kube-api-access-9d48t" pod "51c49a39-5522-4cec-9246-7de8e59d4b9f" (UID: "51c49a39-5522-4cec-9246-7de8e59d4b9f"). InnerVolumeSpecName "kube-api-access-9d48t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:19:22.499271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.499240    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-podinfo" pod "51c49a39-5522-4cec-9246-7de8e59d4b9f" (UID: "51c49a39-5522-4cec-9246-7de8e59d4b9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:19:22.499271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.499253    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-token" pod "51c49a39-5522-4cec-9246-7de8e59d4b9f" (UID: "51c49a39-5522-4cec-9246-7de8e59d4b9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:19:22.585332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.585027    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:22.585332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.585070    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:22.585332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.585086    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d48t\" (UniqueName: \"kubernetes.io/projected/51c49a39-5522-4cec-9246-7de8e59d4b9f-kube-api-access-9d48t\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:22.585332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.585103    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:22.585332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.585117    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51c49a39-5522-4cec-9246-7de8e59d4b9f-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:22.585332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.585131    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:22.585332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.585146    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:22.585332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.585160    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:22.585332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.585174    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51c49a39-5522-4cec-9246-7de8e59d4b9f-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:22.906270 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.902594    4294 generic.go:356] "Generic (PLEG): container finished" podID="51c49a39-5522-4cec-9246-7de8e59d4b9f" containerID="a5a52c2bb72e102fb1330f9a487b8ebdc6688bed91fc90bfff88230064d29b0c" exitCode=0
Nov 02 23:19:22.906270 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.902765    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff" event={"ID":"51c49a39-5522-4cec-9246-7de8e59d4b9f","Type":"ContainerDied","Data":"a5a52c2bb72e102fb1330f9a487b8ebdc6688bed91fc90bfff88230064d29b0c"}
Nov 02 23:19:22.906270 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.902817    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff" event={"ID":"51c49a39-5522-4cec-9246-7de8e59d4b9f","Type":"ContainerDied","Data":"96351999d8992b9eff31cfa37e847961a974372f3cae13f840cadd4219f483ec"}
Nov 02 23:19:22.906270 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.902845    4294 scope.go:122] "RemoveContainer" containerID="a5a52c2bb72e102fb1330f9a487b8ebdc6688bed91fc90bfff88230064d29b0c"
Nov 02 23:19:22.906270 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.903036    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"
Nov 02 23:19:22.917528 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.917505    4294 scope.go:122] "RemoveContainer" containerID="a5a52c2bb72e102fb1330f9a487b8ebdc6688bed91fc90bfff88230064d29b0c"
Nov 02 23:19:22.917866 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:19:22.917842    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a52c2bb72e102fb1330f9a487b8ebdc6688bed91fc90bfff88230064d29b0c\": container with ID starting with a5a52c2bb72e102fb1330f9a487b8ebdc6688bed91fc90bfff88230064d29b0c not found: ID does not exist" containerID="a5a52c2bb72e102fb1330f9a487b8ebdc6688bed91fc90bfff88230064d29b0c"
Nov 02 23:19:22.917963 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:22.917874    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a52c2bb72e102fb1330f9a487b8ebdc6688bed91fc90bfff88230064d29b0c"} err="failed to get container status \"a5a52c2bb72e102fb1330f9a487b8ebdc6688bed91fc90bfff88230064d29b0c\": rpc error: code = NotFound desc = could not find container \"a5a52c2bb72e102fb1330f9a487b8ebdc6688bed91fc90bfff88230064d29b0c\": container with ID starting with a5a52c2bb72e102fb1330f9a487b8ebdc6688bed91fc90bfff88230064d29b0c not found: ID does not exist"
Nov 02 23:19:23.096685 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:23.096646    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"]
Nov 02 23:19:23.099547 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:23.099519    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-6rwff"]
Nov 02 23:19:23.534630 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:23.534594    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 23:19:23.535070 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:23.534646    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh" podUID="5d17d552-d441-4ed5-a5c1-5a4141808868" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 23:19:23.810249 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:23.810218    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 02 23:19:23.810445 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:23.810266    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq" podUID="52a13074-c991-4d08-9bfd-c132ffab4c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 02 23:19:23.914632 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:23.914484    4294 generic.go:356] "Generic (PLEG): container finished" podID="5d17d552-d441-4ed5-a5c1-5a4141808868" containerID="e89c673aef0fa32412334bcf32446db1b36890b9240345263467d31d72fa3f7b" exitCode=0
Nov 02 23:19:23.914632 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:23.914579    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh" event={"ID":"5d17d552-d441-4ed5-a5c1-5a4141808868","Type":"ContainerDied","Data":"e89c673aef0fa32412334bcf32446db1b36890b9240345263467d31d72fa3f7b"}
Nov 02 23:19:23.916817 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:23.916314    4294 generic.go:356] "Generic (PLEG): container finished" podID="67334c5b-8d17-478e-8248-0762c0e1653c" containerID="0824e99f6eb8e9ee9e5ae119af94a660358274fb1d6f8228ed5d4ea2707d2556" exitCode=0
Nov 02 23:19:23.916817 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:23.916430    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp" event={"ID":"67334c5b-8d17-478e-8248-0762c0e1653c","Type":"ContainerDied","Data":"0824e99f6eb8e9ee9e5ae119af94a660358274fb1d6f8228ed5d4ea2707d2556"}
Nov 02 23:19:23.919843 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:23.919809    4294 generic.go:356] "Generic (PLEG): container finished" podID="9d525079-fdb8-4c0e-bb38-2567fcf4a49a" containerID="fbd56b7432201d2d341e246841a0cccaa25826338200dcdaabf3688945d3b21b" exitCode=0
Nov 02 23:19:23.919949 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:23.919931    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp" event={"ID":"9d525079-fdb8-4c0e-bb38-2567fcf4a49a","Type":"ContainerDied","Data":"fbd56b7432201d2d341e246841a0cccaa25826338200dcdaabf3688945d3b21b"}
Nov 02 23:19:23.920112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:23.920008    4294 scope.go:122] "RemoveContainer" containerID="58c2c1ee3f93ef3829295a5eee14912143845af9968d63c703eea9ac2c5a9311"
Nov 02 23:19:23.922861 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:23.922839    4294 generic.go:356] "Generic (PLEG): container finished" podID="52a13074-c991-4d08-9bfd-c132ffab4c90" containerID="616c8c98911f49a0700c674bd8f5d70b55d3e18d508732769f818a750011cc0d" exitCode=0
Nov 02 23:19:23.922953 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:23.922876    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq" event={"ID":"52a13074-c991-4d08-9bfd-c132ffab4c90","Type":"ContainerDied","Data":"616c8c98911f49a0700c674bd8f5d70b55d3e18d508732769f818a750011cc0d"}
Nov 02 23:19:24.051915 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.042621    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:24.209838 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.209806    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:24.245207 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.245177    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:19:24.247384 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.247362    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:19:24.314649 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.314499    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:19:24.343491 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.343453    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:19:24.419191 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419161    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d17d552-d441-4ed5-a5c1-5a4141808868-kube-api-access-q5w94\" (UniqueName: \"kubernetes.io/projected/5d17d552-d441-4ed5-a5c1-5a4141808868-kube-api-access-q5w94\") pod \"5d17d552-d441-4ed5-a5c1-5a4141808868\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") "
Nov 02 23:19:24.419337 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419200    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-podinfo\") pod \"52a13074-c991-4d08-9bfd-c132ffab4c90\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") "
Nov 02 23:19:24.419337 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419239    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-podinfo\") pod \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") "
Nov 02 23:19:24.419337 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419269    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-workload-socket\") pod \"52a13074-c991-4d08-9bfd-c132ffab4c90\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") "
Nov 02 23:19:24.419337 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419305    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-credential-socket\") pod \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") "
Nov 02 23:19:24.419337 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419325    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52a13074-c991-4d08-9bfd-c132ffab4c90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52a13074-c991-4d08-9bfd-c132ffab4c90-istiod-ca-cert\") pod \"52a13074-c991-4d08-9bfd-c132ffab4c90\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") "
Nov 02 23:19:24.419532 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419347    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-data\") pod \"52a13074-c991-4d08-9bfd-c132ffab4c90\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") "
Nov 02 23:19:24.419532 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419415    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-workload-certs\") pod \"52a13074-c991-4d08-9bfd-c132ffab4c90\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") "
Nov 02 23:19:24.419532 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419443    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-workload-certs\") pod \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") "
Nov 02 23:19:24.419532 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419468    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-podinfo\") pod \"5d17d552-d441-4ed5-a5c1-5a4141808868\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") "
Nov 02 23:19:24.419532 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419488    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-kube-api-access-4z4bd\" (UniqueName: \"kubernetes.io/projected/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-kube-api-access-4z4bd\") pod \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") "
Nov 02 23:19:24.419532 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419518    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-token\" (UniqueName: \"kubernetes.io/projected/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-token\") pod \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") "
Nov 02 23:19:24.419709 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419542    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-token\" (UniqueName: \"kubernetes.io/projected/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-token\") pod \"5d17d552-d441-4ed5-a5c1-5a4141808868\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") "
Nov 02 23:19:24.419709 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419571    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-envoy\") pod \"5d17d552-d441-4ed5-a5c1-5a4141808868\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") "
Nov 02 23:19:24.419709 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419594    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-credential-socket\") pod \"52a13074-c991-4d08-9bfd-c132ffab4c90\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") "
Nov 02 23:19:24.419709 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419630    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-data\") pod \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") "
Nov 02 23:19:24.419709 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419668    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-token\" (UniqueName: \"kubernetes.io/projected/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-token\") pod \"52a13074-c991-4d08-9bfd-c132ffab4c90\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") "
Nov 02 23:19:24.419709 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419692    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-credential-socket\") pod \"5d17d552-d441-4ed5-a5c1-5a4141808868\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") "
Nov 02 23:19:24.419881 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419720    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-data\") pod \"5d17d552-d441-4ed5-a5c1-5a4141808868\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") "
Nov 02 23:19:24.419881 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419753    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-workload-certs\") pod \"5d17d552-d441-4ed5-a5c1-5a4141808868\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") "
Nov 02 23:19:24.419881 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419784    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-workload-socket\") pod \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") "
Nov 02 23:19:24.419881 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419802    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d17d552-d441-4ed5-a5c1-5a4141808868-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d17d552-d441-4ed5-a5c1-5a4141808868-istiod-ca-cert\") pod \"5d17d552-d441-4ed5-a5c1-5a4141808868\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") "
Nov 02 23:19:24.419881 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419845    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a13074-c991-4d08-9bfd-c132ffab4c90-kube-api-access-xhgcx\" (UniqueName: \"kubernetes.io/projected/52a13074-c991-4d08-9bfd-c132ffab4c90-kube-api-access-xhgcx\") pod \"52a13074-c991-4d08-9bfd-c132ffab4c90\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") "
Nov 02 23:19:24.419881 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419870    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istiod-ca-cert\") pod \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") "
Nov 02 23:19:24.420100 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419914    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-workload-socket\") pod \"5d17d552-d441-4ed5-a5c1-5a4141808868\" (UID: \"5d17d552-d441-4ed5-a5c1-5a4141808868\") "
Nov 02 23:19:24.420100 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.419944    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-envoy\") pod \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\" (UID: \"9d525079-fdb8-4c0e-bb38-2567fcf4a49a\") "
Nov 02 23:19:24.420100 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.420028    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-envoy\") pod \"52a13074-c991-4d08-9bfd-c132ffab4c90\" (UID: \"52a13074-c991-4d08-9bfd-c132ffab4c90\") "
Nov 02 23:19:24.421130 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.421093    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-workload-certs" pod "9d525079-fdb8-4c0e-bb38-2567fcf4a49a" (UID: "9d525079-fdb8-4c0e-bb38-2567fcf4a49a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:24.421307 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.421186    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a13074-c991-4d08-9bfd-c132ffab4c90-istiod-ca-cert" pod "52a13074-c991-4d08-9bfd-c132ffab4c90" (UID: "52a13074-c991-4d08-9bfd-c132ffab4c90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:19:24.421672 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.421639    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-workload-certs" pod "52a13074-c991-4d08-9bfd-c132ffab4c90" (UID: "52a13074-c991-4d08-9bfd-c132ffab4c90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:24.421783 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.421701    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-data" pod "52a13074-c991-4d08-9bfd-c132ffab4c90" (UID: "52a13074-c991-4d08-9bfd-c132ffab4c90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:24.422233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.422205    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-workload-socket" pod "52a13074-c991-4d08-9bfd-c132ffab4c90" (UID: "52a13074-c991-4d08-9bfd-c132ffab4c90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:24.422530 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.422407    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-credential-socket" pod "9d525079-fdb8-4c0e-bb38-2567fcf4a49a" (UID: "9d525079-fdb8-4c0e-bb38-2567fcf4a49a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:24.423178 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.422728    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-credential-socket" pod "5d17d552-d441-4ed5-a5c1-5a4141808868" (UID: "5d17d552-d441-4ed5-a5c1-5a4141808868"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:24.423178 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.422846    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-data" pod "5d17d552-d441-4ed5-a5c1-5a4141808868" (UID: "5d17d552-d441-4ed5-a5c1-5a4141808868"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:24.423178 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.422941    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-credential-socket" pod "52a13074-c991-4d08-9bfd-c132ffab4c90" (UID: "52a13074-c991-4d08-9bfd-c132ffab4c90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:24.423178 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.422992    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-workload-certs" pod "5d17d552-d441-4ed5-a5c1-5a4141808868" (UID: "5d17d552-d441-4ed5-a5c1-5a4141808868"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:24.423685 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.423658    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-data" pod "9d525079-fdb8-4c0e-bb38-2567fcf4a49a" (UID: "9d525079-fdb8-4c0e-bb38-2567fcf4a49a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:24.423784 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.423741    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-podinfo" pod "5d17d552-d441-4ed5-a5c1-5a4141808868" (UID: "5d17d552-d441-4ed5-a5c1-5a4141808868"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:19:24.424264 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.423961    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-workload-socket" pod "5d17d552-d441-4ed5-a5c1-5a4141808868" (UID: "5d17d552-d441-4ed5-a5c1-5a4141808868"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:24.424264 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.424125    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-workload-socket" pod "9d525079-fdb8-4c0e-bb38-2567fcf4a49a" (UID: "9d525079-fdb8-4c0e-bb38-2567fcf4a49a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:24.424264 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.424226    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d17d552-d441-4ed5-a5c1-5a4141808868-istiod-ca-cert" pod "5d17d552-d441-4ed5-a5c1-5a4141808868" (UID: "5d17d552-d441-4ed5-a5c1-5a4141808868"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:19:24.424576 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.424298    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istiod-ca-cert" pod "9d525079-fdb8-4c0e-bb38-2567fcf4a49a" (UID: "9d525079-fdb8-4c0e-bb38-2567fcf4a49a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:19:24.426071 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.426037    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-envoy" pod "52a13074-c991-4d08-9bfd-c132ffab4c90" (UID: "52a13074-c991-4d08-9bfd-c132ffab4c90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:24.426210 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.426188    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d17d552-d441-4ed5-a5c1-5a4141808868-kube-api-access-q5w94" pod "5d17d552-d441-4ed5-a5c1-5a4141808868" (UID: "5d17d552-d441-4ed5-a5c1-5a4141808868"). InnerVolumeSpecName "kube-api-access-q5w94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:19:24.426507 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.426451    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-kube-api-access-4z4bd" pod "9d525079-fdb8-4c0e-bb38-2567fcf4a49a" (UID: "9d525079-fdb8-4c0e-bb38-2567fcf4a49a"). InnerVolumeSpecName "kube-api-access-4z4bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:19:24.426733 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.426704    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-token" pod "52a13074-c991-4d08-9bfd-c132ffab4c90" (UID: "52a13074-c991-4d08-9bfd-c132ffab4c90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:19:24.427833 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.427810    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-podinfo" pod "9d525079-fdb8-4c0e-bb38-2567fcf4a49a" (UID: "9d525079-fdb8-4c0e-bb38-2567fcf4a49a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:19:24.427905 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.427851    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-token" pod "5d17d552-d441-4ed5-a5c1-5a4141808868" (UID: "5d17d552-d441-4ed5-a5c1-5a4141808868"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:19:24.428063 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.428028    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-envoy" pod "9d525079-fdb8-4c0e-bb38-2567fcf4a49a" (UID: "9d525079-fdb8-4c0e-bb38-2567fcf4a49a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:24.428170 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.428151    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a13074-c991-4d08-9bfd-c132ffab4c90-kube-api-access-xhgcx" pod "52a13074-c991-4d08-9bfd-c132ffab4c90" (UID: "52a13074-c991-4d08-9bfd-c132ffab4c90"). InnerVolumeSpecName "kube-api-access-xhgcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:19:24.428328 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.428307    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-envoy" pod "5d17d552-d441-4ed5-a5c1-5a4141808868" (UID: "5d17d552-d441-4ed5-a5c1-5a4141808868"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:24.428390 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.428374    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-podinfo" pod "52a13074-c991-4d08-9bfd-c132ffab4c90" (UID: "52a13074-c991-4d08-9bfd-c132ffab4c90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:19:24.428467 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.428445    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-token" pod "9d525079-fdb8-4c0e-bb38-2567fcf4a49a" (UID: "9d525079-fdb8-4c0e-bb38-2567fcf4a49a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:19:24.521470 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.521424    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67334c5b-8d17-478e-8248-0762c0e1653c-istio-token\" (UniqueName: \"kubernetes.io/projected/67334c5b-8d17-478e-8248-0762c0e1653c-istio-token\") pod \"67334c5b-8d17-478e-8248-0762c0e1653c\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") "
Nov 02 23:19:24.521680 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.521552    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67334c5b-8d17-478e-8248-0762c0e1653c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67334c5b-8d17-478e-8248-0762c0e1653c-istiod-ca-cert\") pod \"67334c5b-8d17-478e-8248-0762c0e1653c\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") "
Nov 02 23:19:24.521680 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.521613    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67334c5b-8d17-478e-8248-0762c0e1653c-kube-api-access-nnvzn\" (UniqueName: \"kubernetes.io/projected/67334c5b-8d17-478e-8248-0762c0e1653c-kube-api-access-nnvzn\") pod \"67334c5b-8d17-478e-8248-0762c0e1653c\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") "
Nov 02 23:19:24.521680 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.521649    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-istio-data\") pod \"67334c5b-8d17-478e-8248-0762c0e1653c\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") "
Nov 02 23:19:24.521848 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.521684    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-workload-certs\") pod \"67334c5b-8d17-478e-8248-0762c0e1653c\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") "
Nov 02 23:19:24.521922 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.521842    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67334c5b-8d17-478e-8248-0762c0e1653c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67334c5b-8d17-478e-8248-0762c0e1653c-istio-podinfo\") pod \"67334c5b-8d17-478e-8248-0762c0e1653c\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") "
Nov 02 23:19:24.521922 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.521902    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-workload-socket\") pod \"67334c5b-8d17-478e-8248-0762c0e1653c\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") "
Nov 02 23:19:24.522041 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.521932    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-istio-envoy\") pod \"67334c5b-8d17-478e-8248-0762c0e1653c\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") "
Nov 02 23:19:24.522041 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.522023    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-credential-socket\") pod \"67334c5b-8d17-478e-8248-0762c0e1653c\" (UID: \"67334c5b-8d17-478e-8248-0762c0e1653c\") "
Nov 02 23:19:24.522715 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.522688    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.522837 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.522720    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52a13074-c991-4d08-9bfd-c132ffab4c90-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.522837 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.522735    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.522837 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.522752    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.522837 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.522768    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.522837 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.522782    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.522837 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.522797    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z4bd\" (UniqueName: \"kubernetes.io/projected/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-kube-api-access-4z4bd\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.522837 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.522811    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.522837 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.522825    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.522837 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.522837    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.523304 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.522851    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.523304 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.522865    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.523304 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.522878    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.523304 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.522892    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.523304 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.522905    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.523304 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.522919    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.523304 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.522936    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.523304 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.522949    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d17d552-d441-4ed5-a5c1-5a4141808868-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.523304 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.522963    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhgcx\" (UniqueName: \"kubernetes.io/projected/52a13074-c991-4d08-9bfd-c132ffab4c90-kube-api-access-xhgcx\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.523757 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.523326    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-istio-data" pod "67334c5b-8d17-478e-8248-0762c0e1653c" (UID: "67334c5b-8d17-478e-8248-0762c0e1653c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:24.523757 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.523696    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.523757 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.523721    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d17d552-d441-4ed5-a5c1-5a4141808868-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.523757 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.523736    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.523757 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.523751    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.524021 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.523766    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5w94\" (UniqueName: \"kubernetes.io/projected/5d17d552-d441-4ed5-a5c1-5a4141808868-kube-api-access-q5w94\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.524021 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.523781    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52a13074-c991-4d08-9bfd-c132ffab4c90-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.524021 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.523794    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d525079-fdb8-4c0e-bb38-2567fcf4a49a-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.524021 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.523808    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52a13074-c991-4d08-9bfd-c132ffab4c90-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.524233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.524085    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-credential-socket" pod "67334c5b-8d17-478e-8248-0762c0e1653c" (UID: "67334c5b-8d17-478e-8248-0762c0e1653c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:24.524311 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.524287    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-workload-certs" pod "67334c5b-8d17-478e-8248-0762c0e1653c" (UID: "67334c5b-8d17-478e-8248-0762c0e1653c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:24.525320 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.525292    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-workload-socket" pod "67334c5b-8d17-478e-8248-0762c0e1653c" (UID: "67334c5b-8d17-478e-8248-0762c0e1653c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:24.525631 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.525609    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67334c5b-8d17-478e-8248-0762c0e1653c-istiod-ca-cert" pod "67334c5b-8d17-478e-8248-0762c0e1653c" (UID: "67334c5b-8d17-478e-8248-0762c0e1653c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:19:24.576778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.560728    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51c49a39-5522-4cec-9246-7de8e59d4b9f" path="/var/lib/kubelet/pods/51c49a39-5522-4cec-9246-7de8e59d4b9f/volumes"
Nov 02 23:19:24.629265 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.629183    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.629265 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.629215    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.629670 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.629601    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67334c5b-8d17-478e-8248-0762c0e1653c-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.629670 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.629634    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.629670 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.629649    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.634815 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.634766    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67334c5b-8d17-478e-8248-0762c0e1653c-istio-podinfo" pod "67334c5b-8d17-478e-8248-0762c0e1653c" (UID: "67334c5b-8d17-478e-8248-0762c0e1653c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:19:24.636259 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.636209    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67334c5b-8d17-478e-8248-0762c0e1653c-istio-token" pod "67334c5b-8d17-478e-8248-0762c0e1653c" (UID: "67334c5b-8d17-478e-8248-0762c0e1653c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:19:24.642814 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.639196    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67334c5b-8d17-478e-8248-0762c0e1653c-kube-api-access-nnvzn" pod "67334c5b-8d17-478e-8248-0762c0e1653c" (UID: "67334c5b-8d17-478e-8248-0762c0e1653c"). InnerVolumeSpecName "kube-api-access-nnvzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:19:24.676964 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.650098    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-istio-envoy" pod "67334c5b-8d17-478e-8248-0762c0e1653c" (UID: "67334c5b-8d17-478e-8248-0762c0e1653c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:19:24.731808 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.731772    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67334c5b-8d17-478e-8248-0762c0e1653c-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.732046 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.732032    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67334c5b-8d17-478e-8248-0762c0e1653c-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.732112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.732105    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67334c5b-8d17-478e-8248-0762c0e1653c-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.732189 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.732179    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnvzn\" (UniqueName: \"kubernetes.io/projected/67334c5b-8d17-478e-8248-0762c0e1653c-kube-api-access-nnvzn\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:24.931484 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.931385    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq" event={"ID":"52a13074-c991-4d08-9bfd-c132ffab4c90","Type":"ContainerDied","Data":"19e47e8933404ba45061021cb12a1a7343af669a8c53bbb33d8569b363e31a14"}
Nov 02 23:19:24.931662 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.931484    4294 scope.go:122] "RemoveContainer" containerID="616c8c98911f49a0700c674bd8f5d70b55d3e18d508732769f818a750011cc0d"
Nov 02 23:19:24.931662 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.931627    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"
Nov 02 23:19:24.933437 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.933415    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"
Nov 02 23:19:24.933830 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.933415    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh" event={"ID":"5d17d552-d441-4ed5-a5c1-5a4141808868","Type":"ContainerDied","Data":"38f6ff74016eba7d9abd151d6965be331f9dfe871ce121b46b9e607e40d069f7"}
Nov 02 23:19:24.935079 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.934967    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp" event={"ID":"67334c5b-8d17-478e-8248-0762c0e1653c","Type":"ContainerDied","Data":"c3b9cf0780080e552e9cf300ada619af2aab55f9ad7a1a05872f03e8774aca76"}
Nov 02 23:19:24.935079 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.934999    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"
Nov 02 23:19:24.936420 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.936396    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp" event={"ID":"9d525079-fdb8-4c0e-bb38-2567fcf4a49a","Type":"ContainerDied","Data":"a5f43731456fd3b6e57b65057b5773bc66a92a949f206dac66919824f75c7bb4"}
Nov 02 23:19:24.936488 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.936477    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"
Nov 02 23:19:24.946389 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.946374    4294 scope.go:122] "RemoveContainer" containerID="e89c673aef0fa32412334bcf32446db1b36890b9240345263467d31d72fa3f7b"
Nov 02 23:19:24.978006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.976353    4294 scope.go:122] "RemoveContainer" containerID="0824e99f6eb8e9ee9e5ae119af94a660358274fb1d6f8228ed5d4ea2707d2556"
Nov 02 23:19:24.988853 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:24.988830    4294 scope.go:122] "RemoveContainer" containerID="fbd56b7432201d2d341e246841a0cccaa25826338200dcdaabf3688945d3b21b"
Nov 02 23:19:25.090270 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:25.090229    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"]
Nov 02 23:19:25.095857 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:25.095825    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-r6lpp"]
Nov 02 23:19:25.203758 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:25.203393    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"]
Nov 02 23:19:25.206751 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:25.206724    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-8fllq"]
Nov 02 23:19:25.307920 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:25.307886    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"]
Nov 02 23:19:25.310861 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:25.310841    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-5hhvp"]
Nov 02 23:19:25.434087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:25.434054    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"]
Nov 02 23:19:25.441265 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:25.441233    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-l4wkh"]
Nov 02 23:19:26.550050 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:26.550022    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52a13074-c991-4d08-9bfd-c132ffab4c90" path="/var/lib/kubelet/pods/52a13074-c991-4d08-9bfd-c132ffab4c90/volumes"
Nov 02 23:19:26.550660 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:26.550645    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d17d552-d441-4ed5-a5c1-5a4141808868" path="/var/lib/kubelet/pods/5d17d552-d441-4ed5-a5c1-5a4141808868/volumes"
Nov 02 23:19:26.551107 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:26.551093    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67334c5b-8d17-478e-8248-0762c0e1653c" path="/var/lib/kubelet/pods/67334c5b-8d17-478e-8248-0762c0e1653c/volumes"
Nov 02 23:19:26.551535 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:26.551517    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d525079-fdb8-4c0e-bb38-2567fcf4a49a" path="/var/lib/kubelet/pods/9d525079-fdb8-4c0e-bb38-2567fcf4a49a/volumes"
Nov 02 23:19:27.253310 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:27.253276    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:30.469123 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:30.469087    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:32.475603 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:32.475568    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:34.346686 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:34.346649    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:34.942283 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:34.942249    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:35.425767 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:35.425730    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:37.711711 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:37.711663    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:38.994792 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:38.994760    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:39.170732 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:39.170688    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:39.345904 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:39.345817    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:41.751105 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:19:41.751062    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:19:42.183879 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.183801    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:42.346331 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.346298    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:42.557595 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.557564    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm"]
Nov 02 23:19:42.558242 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558223    4294 cpu_manager.go:414] "containerMap: removing container" podUID="5d17d552-d441-4ed5-a5c1-5a4141808868" containerName="istio-proxy"
Nov 02 23:19:42.558371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558242    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d17d552-d441-4ed5-a5c1-5a4141808868" containerName="istio-proxy"
Nov 02 23:19:42.558371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558271    4294 cpu_manager.go:414] "containerMap: removing container" podUID="9d525079-fdb8-4c0e-bb38-2567fcf4a49a" containerName="istio-proxy"
Nov 02 23:19:42.558371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558277    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d525079-fdb8-4c0e-bb38-2567fcf4a49a" containerName="istio-proxy"
Nov 02 23:19:42.558371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558295    4294 cpu_manager.go:414] "containerMap: removing container" podUID="67334c5b-8d17-478e-8248-0762c0e1653c" containerName="istio-proxy"
Nov 02 23:19:42.558371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558301    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67334c5b-8d17-478e-8248-0762c0e1653c" containerName="istio-proxy"
Nov 02 23:19:42.558371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558312    4294 cpu_manager.go:414] "containerMap: removing container" podUID="51c49a39-5522-4cec-9246-7de8e59d4b9f" containerName="istio-proxy"
Nov 02 23:19:42.558371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558316    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51c49a39-5522-4cec-9246-7de8e59d4b9f" containerName="istio-proxy"
Nov 02 23:19:42.558371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558328    4294 cpu_manager.go:414] "containerMap: removing container" podUID="52a13074-c991-4d08-9bfd-c132ffab4c90" containerName="istio-proxy"
Nov 02 23:19:42.558371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558333    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a13074-c991-4d08-9bfd-c132ffab4c90" containerName="istio-proxy"
Nov 02 23:19:42.558371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558347    4294 cpu_manager.go:414] "containerMap: removing container" podUID="09f13590-7f6c-4c1c-90c2-b59751b42720" containerName="istio-proxy"
Nov 02 23:19:42.558371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558352    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f13590-7f6c-4c1c-90c2-b59751b42720" containerName="istio-proxy"
Nov 02 23:19:42.558371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558361    4294 cpu_manager.go:414] "containerMap: removing container" podUID="9d525079-fdb8-4c0e-bb38-2567fcf4a49a" containerName="istio-proxy"
Nov 02 23:19:42.558371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558366    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d525079-fdb8-4c0e-bb38-2567fcf4a49a" containerName="istio-proxy"
Nov 02 23:19:42.558371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558376    4294 cpu_manager.go:414] "containerMap: removing container" podUID="50895b59-7759-4a46-aa8f-a1d691c255de" containerName="istio-proxy"
Nov 02 23:19:42.558787 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558381    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50895b59-7759-4a46-aa8f-a1d691c255de" containerName="istio-proxy"
Nov 02 23:19:42.558787 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558463    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d525079-fdb8-4c0e-bb38-2567fcf4a49a" containerName="istio-proxy"
Nov 02 23:19:42.558787 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558471    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="50895b59-7759-4a46-aa8f-a1d691c255de" containerName="istio-proxy"
Nov 02 23:19:42.558787 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558480    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d17d552-d441-4ed5-a5c1-5a4141808868" containerName="istio-proxy"
Nov 02 23:19:42.558787 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558486    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d525079-fdb8-4c0e-bb38-2567fcf4a49a" containerName="istio-proxy"
Nov 02 23:19:42.558787 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558493    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="67334c5b-8d17-478e-8248-0762c0e1653c" containerName="istio-proxy"
Nov 02 23:19:42.558787 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558498    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="51c49a39-5522-4cec-9246-7de8e59d4b9f" containerName="istio-proxy"
Nov 02 23:19:42.558787 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558513    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a13074-c991-4d08-9bfd-c132ffab4c90" containerName="istio-proxy"
Nov 02 23:19:42.558787 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.558520    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f13590-7f6c-4c1c-90c2-b59751b42720" containerName="istio-proxy"
Nov 02 23:19:42.562781 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.562762    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm"
Nov 02 23:19:42.614508 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.614472    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm"]
Nov 02 23:19:42.659687 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.659656    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a69f65-dockercfg-x9jzb\""
Nov 02 23:19:42.660071 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.660042    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a69f65\""
Nov 02 23:19:42.706299 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.706263    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbhqv\" (UniqueName: \"kubernetes.io/projected/443dc393-32b0-4ca3-be71-7d6f77747162-kube-api-access-zbhqv\") pod \"payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm\" (UID: \"443dc393-32b0-4ca3-be71-7d6f77747162\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm"
Nov 02 23:19:42.707838 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.707808    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/443dc393-32b0-4ca3-be71-7d6f77747162-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm\" (UID: \"443dc393-32b0-4ca3-be71-7d6f77747162\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm"
Nov 02 23:19:42.725775 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.725721    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88"]
Nov 02 23:19:42.731164 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.731134    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88"
Nov 02 23:19:42.772192 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.772162    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88"]
Nov 02 23:19:42.809898 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.809804    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbhqv\" (UniqueName: \"kubernetes.io/projected/443dc393-32b0-4ca3-be71-7d6f77747162-kube-api-access-zbhqv\") pod \"payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm\" (UID: \"443dc393-32b0-4ca3-be71-7d6f77747162\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm"
Nov 02 23:19:42.810086 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.809999    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/443dc393-32b0-4ca3-be71-7d6f77747162-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm\" (UID: \"443dc393-32b0-4ca3-be71-7d6f77747162\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm"
Nov 02 23:19:42.810846 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.810825    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/443dc393-32b0-4ca3-be71-7d6f77747162-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm\" (UID: \"443dc393-32b0-4ca3-be71-7d6f77747162\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm"
Nov 02 23:19:42.864648 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.864603    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbhqv\" (UniqueName: \"kubernetes.io/projected/443dc393-32b0-4ca3-be71-7d6f77747162-kube-api-access-zbhqv\") pod \"payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm\" (UID: \"443dc393-32b0-4ca3-be71-7d6f77747162\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm"
Nov 02 23:19:42.875922 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.875889    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm"
Nov 02 23:19:42.911579 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.911539    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4qbp\" (UniqueName: \"kubernetes.io/projected/87085fd9-8b23-4f20-8819-562321e3b11e-kube-api-access-x4qbp\") pod \"payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88\" (UID: \"87085fd9-8b23-4f20-8819-562321e3b11e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88"
Nov 02 23:19:42.911796 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:42.911693    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87085fd9-8b23-4f20-8819-562321e3b11e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88\" (UID: \"87085fd9-8b23-4f20-8819-562321e3b11e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88"
Nov 02 23:19:43.012606 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:43.012565    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4qbp\" (UniqueName: \"kubernetes.io/projected/87085fd9-8b23-4f20-8819-562321e3b11e-kube-api-access-x4qbp\") pod \"payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88\" (UID: \"87085fd9-8b23-4f20-8819-562321e3b11e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88"
Nov 02 23:19:43.012807 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:43.012712    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87085fd9-8b23-4f20-8819-562321e3b11e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88\" (UID: \"87085fd9-8b23-4f20-8819-562321e3b11e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88"
Nov 02 23:19:43.013417 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:43.013396    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87085fd9-8b23-4f20-8819-562321e3b11e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88\" (UID: \"87085fd9-8b23-4f20-8819-562321e3b11e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88"
Nov 02 23:19:43.067886 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:43.067814    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4qbp\" (UniqueName: \"kubernetes.io/projected/87085fd9-8b23-4f20-8819-562321e3b11e-kube-api-access-x4qbp\") pod \"payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88\" (UID: \"87085fd9-8b23-4f20-8819-562321e3b11e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88"
Nov 02 23:19:43.197285 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:43.197245    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:43.249314 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:43.249282    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm"]
Nov 02 23:19:43.346582 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:43.346502    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88"
Nov 02 23:19:43.717404 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:43.717368    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88"]
Nov 02 23:19:43.717958 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:19:43.717926    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87085fd9_8b23_4f20_8819_562321e3b11e.slice/crio-8a2e1b454042832c6428124c4164dc9293c23513ad5d15482bbfeb3732610c38 WatchSource:0}: Error finding container 8a2e1b454042832c6428124c4164dc9293c23513ad5d15482bbfeb3732610c38: Status 404 returned error can't find the container with id 8a2e1b454042832c6428124c4164dc9293c23513ad5d15482bbfeb3732610c38
Nov 02 23:19:44.089994 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:44.089949    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88" event={"ID":"87085fd9-8b23-4f20-8819-562321e3b11e","Type":"ContainerStarted","Data":"669ddfbba1a0cf388e3300122568d3004f617fb26bd7ff50a441246ca2594588"}
Nov 02 23:19:44.090492 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:44.090049    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88" event={"ID":"87085fd9-8b23-4f20-8819-562321e3b11e","Type":"ContainerStarted","Data":"8a2e1b454042832c6428124c4164dc9293c23513ad5d15482bbfeb3732610c38"}
Nov 02 23:19:44.090492 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:44.090167    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88"
Nov 02 23:19:44.091648 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:44.091621    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm" event={"ID":"443dc393-32b0-4ca3-be71-7d6f77747162","Type":"ContainerStarted","Data":"2713a7c62b6f441430d90c10a1d39f70c34a83a12981e9a2e3138ff6d0ba80b0"}
Nov 02 23:19:44.091768 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:44.091662    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm" event={"ID":"443dc393-32b0-4ca3-be71-7d6f77747162","Type":"ContainerStarted","Data":"ff0537bd2993c59d2d8fe3058295f9aa40d80e3cca8d546be6c9facbcec08c6d"}
Nov 02 23:19:44.091768 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:44.091753    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm"
Nov 02 23:19:44.198521 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:44.198469    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88" podStartSLOduration=2.198454791 podStartE2EDuration="2.198454791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:44.197467084 +0000 UTC m=+1623.765821871" watchObservedRunningTime="2026-11-02 23:19:44.198454791 +0000 UTC m=+1623.766809579"
Nov 02 23:19:44.310023 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:44.306075    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm" podStartSLOduration=2.30605457 podStartE2EDuration="2.30605457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:19:44.305075277 +0000 UTC m=+1623.873430065" watchObservedRunningTime="2026-11-02 23:19:44.30605457 +0000 UTC m=+1623.874409358"
Nov 02 23:19:46.061370 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:46.061335    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:47.204550 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:47.204514    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:48.478818 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:48.478783    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:52.699948 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:52.699903    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8"]
Nov 02 23:19:52.971334 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:52.971246    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:53.137383 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:53.137347    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:53.356203 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:53.356159    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:53.574336 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:53.574303    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:53.729856 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:53.729771    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:55.100745 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:55.100706    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88"
Nov 02 23:19:55.101180 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:55.100780    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm"
Nov 02 23:19:57.700792 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:57.700746    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8" podUID="6a5dd45a-b902-476e-853f-7e09438bc122" containerName="payload-pre-processing" containerID="cri-o://091e0561ba36d0d4b915872171ae734e70a64d49a109a3c41db8cc347403711b" gracePeriod=25
Nov 02 23:19:57.973067 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:57.973038    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8"
Nov 02 23:19:58.132376 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.132344    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5dd45a-b902-476e-853f-7e09438bc122-kube-api-access-htwhl\" (UniqueName: \"kubernetes.io/projected/6a5dd45a-b902-476e-853f-7e09438bc122-kube-api-access-htwhl\") pod \"6a5dd45a-b902-476e-853f-7e09438bc122\" (UID: \"6a5dd45a-b902-476e-853f-7e09438bc122\") "
Nov 02 23:19:58.132551 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.132432    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a5dd45a-b902-476e-853f-7e09438bc122-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5dd45a-b902-476e-853f-7e09438bc122-plugins-config-volume\") pod \"6a5dd45a-b902-476e-853f-7e09438bc122\" (UID: \"6a5dd45a-b902-476e-853f-7e09438bc122\") "
Nov 02 23:19:58.132891 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.132858    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5dd45a-b902-476e-853f-7e09438bc122-plugins-config-volume" pod "6a5dd45a-b902-476e-853f-7e09438bc122" (UID: "6a5dd45a-b902-476e-853f-7e09438bc122"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:19:58.133053 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.133031    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5dd45a-b902-476e-853f-7e09438bc122-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:58.134641 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.134617    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5dd45a-b902-476e-853f-7e09438bc122-kube-api-access-htwhl" pod "6a5dd45a-b902-476e-853f-7e09438bc122" (UID: "6a5dd45a-b902-476e-853f-7e09438bc122"). InnerVolumeSpecName "kube-api-access-htwhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:19:58.179553 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.179521    4294 generic.go:356] "Generic (PLEG): container finished" podID="6a5dd45a-b902-476e-853f-7e09438bc122" containerID="091e0561ba36d0d4b915872171ae734e70a64d49a109a3c41db8cc347403711b" exitCode=0
Nov 02 23:19:58.179730 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.179574    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8"
Nov 02 23:19:58.179730 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.179607    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8" event={"ID":"6a5dd45a-b902-476e-853f-7e09438bc122","Type":"ContainerDied","Data":"091e0561ba36d0d4b915872171ae734e70a64d49a109a3c41db8cc347403711b"}
Nov 02 23:19:58.179730 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.179672    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8" event={"ID":"6a5dd45a-b902-476e-853f-7e09438bc122","Type":"ContainerDied","Data":"3d303db536063061f237d686a0d9017e474a270255da5733aacff538b850ffae"}
Nov 02 23:19:58.179730 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.179700    4294 scope.go:122] "RemoveContainer" containerID="091e0561ba36d0d4b915872171ae734e70a64d49a109a3c41db8cc347403711b"
Nov 02 23:19:58.190188 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.190172    4294 scope.go:122] "RemoveContainer" containerID="091e0561ba36d0d4b915872171ae734e70a64d49a109a3c41db8cc347403711b"
Nov 02 23:19:58.190517 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:19:58.190496    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"091e0561ba36d0d4b915872171ae734e70a64d49a109a3c41db8cc347403711b\": container with ID starting with 091e0561ba36d0d4b915872171ae734e70a64d49a109a3c41db8cc347403711b not found: ID does not exist" containerID="091e0561ba36d0d4b915872171ae734e70a64d49a109a3c41db8cc347403711b"
Nov 02 23:19:58.190561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.190525    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"091e0561ba36d0d4b915872171ae734e70a64d49a109a3c41db8cc347403711b"} err="failed to get container status \"091e0561ba36d0d4b915872171ae734e70a64d49a109a3c41db8cc347403711b\": rpc error: code = NotFound desc = could not find container \"091e0561ba36d0d4b915872171ae734e70a64d49a109a3c41db8cc347403711b\": container with ID starting with 091e0561ba36d0d4b915872171ae734e70a64d49a109a3c41db8cc347403711b not found: ID does not exist"
Nov 02 23:19:58.234192 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.234113    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htwhl\" (UniqueName: \"kubernetes.io/projected/6a5dd45a-b902-476e-853f-7e09438bc122-kube-api-access-htwhl\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:19:58.339239 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.339200    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8"]
Nov 02 23:19:58.342376 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.342346    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ede388-59dd8567-f4vn8"]
Nov 02 23:19:58.561001 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.555766    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a5dd45a-b902-476e-853f-7e09438bc122" path="/var/lib/kubelet/pods/6a5dd45a-b902-476e-853f-7e09438bc122/volumes"
Nov 02 23:19:58.842714 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.842622    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"]
Nov 02 23:19:58.844095 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.844064    4294 cpu_manager.go:414] "containerMap: removing container" podUID="6a5dd45a-b902-476e-853f-7e09438bc122" containerName="payload-pre-processing"
Nov 02 23:19:58.844095 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.844095    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5dd45a-b902-476e-853f-7e09438bc122" containerName="payload-pre-processing"
Nov 02 23:19:58.844266 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.844246    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5dd45a-b902-476e-853f-7e09438bc122" containerName="payload-pre-processing"
Nov 02 23:19:58.850830 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.849757    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:58.878561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.878522    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"]
Nov 02 23:19:58.897010 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.884932    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:58.903391 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.903345    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"]
Nov 02 23:19:58.946711 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.946677    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"]
Nov 02 23:19:58.960073 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.960040    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:58.960272 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.960192    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:58.960272 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.960225    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98lfv\" (UniqueName: \"kubernetes.io/projected/d080b437-c462-4b4d-8f36-575abce2a2f1-kube-api-access-98lfv\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:58.960272 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.960257    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:58.960439 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.960313    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:58.960439 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.960414    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:58.960644 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.960586    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d080b437-c462-4b4d-8f36-575abce2a2f1-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:58.960773 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.960648    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:58.960773 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.960684    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:58.970917 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:58.970864    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:19:59.062146 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.062105    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d080b437-c462-4b4d-8f36-575abce2a2f1-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:59.062364 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.062214    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.062364 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.062307    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.062364 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.062336    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:59.062364 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.062360    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07652258-9b4f-41e6-bcaf-54ee4c68049d-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.062561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.062385    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:59.062561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.062462    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.062561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.062515    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.062561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.062556    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:59.062742 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.062634    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.062742 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.062712    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:59.062742 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.062729    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:59.062885 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.062794    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98lfv\" (UniqueName: \"kubernetes.io/projected/d080b437-c462-4b4d-8f36-575abce2a2f1-kube-api-access-98lfv\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:59.062885 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.062821    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.062885 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.062860    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:59.063258 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.063227    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:59.063407 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.063265    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:59.063407 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.062973    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d080b437-c462-4b4d-8f36-575abce2a2f1-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:59.063602 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.063360    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmlmt\" (UniqueName: \"kubernetes.io/projected/07652258-9b4f-41e6-bcaf-54ee4c68049d-kube-api-access-cmlmt\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.063709 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.063545    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:59.063709 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.063703    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:59.063870 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.063760    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.064018 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.063996    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:59.065479 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.065451    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:59.065609 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.065588    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:59.117603 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.117526    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:59.117811 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.117785    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98lfv\" (UniqueName: \"kubernetes.io/projected/d080b437-c462-4b4d-8f36-575abce2a2f1-kube-api-access-98lfv\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:59.165334 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.165294    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmlmt\" (UniqueName: \"kubernetes.io/projected/07652258-9b4f-41e6-bcaf-54ee4c68049d-kube-api-access-cmlmt\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.165549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.165402    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.165549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.165451    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.165549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.165493    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.165549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.165523    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07652258-9b4f-41e6-bcaf-54ee4c68049d-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.165763 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.165549    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.165763 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.165586    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.165763 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.165669    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.165912 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.165763    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.166069 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.166043    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-credential-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.166255 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.166237    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-workload-certs\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.166389 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.166302    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07652258-9b4f-41e6-bcaf-54ee4c68049d-istiod-ca-cert\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.166511 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.166484    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-data\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.166644 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.166584    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-workload-socket\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.168364 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.168338    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-envoy\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.168476 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.168377    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-podinfo\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.178609 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.178571    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:19:59.220039 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.219961    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-token\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.220039 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.220026    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmlmt\" (UniqueName: \"kubernetes.io/projected/07652258-9b4f-41e6-bcaf-54ee4c68049d-kube-api-access-cmlmt\") pod \"e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") " pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.237172 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.237136    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:19:59.601229 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.601190    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"]
Nov 02 23:19:59.640587 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:19:59.640550    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd080b437_c462_4b4d_8f36_575abce2a2f1.slice/crio-51125c66bf9ca5a4539a83d6398db212d69b2ec2bd2cf77adcc4aa2d85d7d8b8 WatchSource:0}: Error finding container 51125c66bf9ca5a4539a83d6398db212d69b2ec2bd2cf77adcc4aa2d85d7d8b8: Status 404 returned error can't find the container with id 51125c66bf9ca5a4539a83d6398db212d69b2ec2bd2cf77adcc4aa2d85d7d8b8
Nov 02 23:19:59.642798 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.642767    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:19:59.642887 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.642837    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:19:59.642887 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.642868    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:19:59.652448 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.652414    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"]
Nov 02 23:19:59.654741 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:19:59.654713    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07652258_9b4f_41e6_bcaf_54ee4c68049d.slice/crio-c81047c5e25f3a545ae67f4bf4dac2a80c81ff1cc89b6e068200429082dc9227 WatchSource:0}: Error finding container c81047c5e25f3a545ae67f4bf4dac2a80c81ff1cc89b6e068200429082dc9227: Status 404 returned error can't find the container with id c81047c5e25f3a545ae67f4bf4dac2a80c81ff1cc89b6e068200429082dc9227
Nov 02 23:19:59.657639 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.657601    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:19:59.657739 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.657672    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:19:59.657739 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.657711    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:19:59.789682 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:19:59.789479    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:00.202375 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:00.202333    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2" event={"ID":"d080b437-c462-4b4d-8f36-575abce2a2f1","Type":"ContainerStarted","Data":"33b21398550ab7003b3331e18cd1b21fa388264a87e07228dbbdcc2954374fa5"}
Nov 02 23:20:00.203041 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:00.202973    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2" event={"ID":"d080b437-c462-4b4d-8f36-575abce2a2f1","Type":"ContainerStarted","Data":"51125c66bf9ca5a4539a83d6398db212d69b2ec2bd2cf77adcc4aa2d85d7d8b8"}
Nov 02 23:20:00.206701 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:00.206654    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth" event={"ID":"07652258-9b4f-41e6-bcaf-54ee4c68049d","Type":"ContainerStarted","Data":"c78e8f83a0b84131e9a708d270ea545b8d6850c3c264742b640606970ef2696c"}
Nov 02 23:20:00.206943 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:00.206926    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth" event={"ID":"07652258-9b4f-41e6-bcaf-54ee4c68049d","Type":"ContainerStarted","Data":"c81047c5e25f3a545ae67f4bf4dac2a80c81ff1cc89b6e068200429082dc9227"}
Nov 02 23:20:00.238208 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:00.238077    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:20:00.239405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:00.239375    4294 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 02 23:20:00.239566 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:00.239432    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth" podUID="07652258-9b4f-41e6-bcaf-54ee4c68049d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 02 23:20:00.320406 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:00.320350    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2" podStartSLOduration=2.320336161 podStartE2EDuration="2.320336161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:20:00.317903433 +0000 UTC m=+1639.886258233" watchObservedRunningTime="2026-11-02 23:20:00.320336161 +0000 UTC m=+1639.888690949"
Nov 02 23:20:00.439994 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:00.439931    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth" podStartSLOduration=2.439916304 podStartE2EDuration="2.439916304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:20:00.437006672 +0000 UTC m=+1640.005361458" watchObservedRunningTime="2026-11-02 23:20:00.439916304 +0000 UTC m=+1640.008271092"
Nov 02 23:20:01.192003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:01.188554    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:20:01.192003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:01.188737    4294 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 02 23:20:01.192003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:01.188788    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2" podUID="d080b437-c462-4b4d-8f36-575abce2a2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 02 23:20:01.238790 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:01.238753    4294 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 02 23:20:01.239213 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:01.238818    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth" podUID="07652258-9b4f-41e6-bcaf-54ee4c68049d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 02 23:20:02.183476 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:02.183433    4294 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 02 23:20:02.183888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:02.183853    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2" podUID="d080b437-c462-4b4d-8f36-575abce2a2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 02 23:20:02.238118 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:02.238083    4294 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 02 23:20:02.238383 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:02.238360    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth" podUID="07652258-9b4f-41e6-bcaf-54ee4c68049d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 02 23:20:03.183004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:03.182437    4294 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 02 23:20:03.183004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:03.182508    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2" podUID="d080b437-c462-4b4d-8f36-575abce2a2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 02 23:20:03.238556 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:03.238514    4294 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 02 23:20:03.238732 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:03.238580    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth" podUID="07652258-9b4f-41e6-bcaf-54ee4c68049d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 02 23:20:04.141731 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:04.141692    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:04.189006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:04.180289    4294 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 02 23:20:04.189006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:04.180355    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2" podUID="d080b437-c462-4b4d-8f36-575abce2a2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 02 23:20:04.238613 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:04.238576    4294 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 02 23:20:04.238792 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:04.238634    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth" podUID="07652258-9b4f-41e6-bcaf-54ee4c68049d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 02 23:20:05.193395 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:05.193358    4294 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 02 23:20:05.194015 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:05.193963    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2" podUID="d080b437-c462-4b4d-8f36-575abce2a2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 02 23:20:05.237771 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:05.237743    4294 patch_prober.go:29] interesting pod/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 02 23:20:05.238039 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:05.238017    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth" podUID="07652258-9b4f-41e6-bcaf-54ee4c68049d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 02 23:20:06.284574 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:06.284527    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:20:06.285721 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:06.285694    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:20:06.285925 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:06.285910    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:20:06.286289 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:06.286271    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:20:06.286868 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:06.286846    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:20:06.287373 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:06.287343    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:20:07.505150 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:07.505117    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:09.011435 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:09.011401    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:11.749631 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:20:11.749593    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:20:12.550961 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:12.550929    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:13.995493 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:13.995444    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:18.765610 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:18.765578    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:18.937629 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:18.937597    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:19.099030 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:19.098947    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:23.285622 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:23.285585    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:25.078064 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:25.078028    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:30.149583 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:30.149544    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:30.864732 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:30.864697    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:31.449154 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:31.449116    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:31.646290 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:31.646258    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:33.045671 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:33.045414    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:33.507019 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:33.506961    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:34.267761 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:34.267724    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:35.146189 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:35.146155    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:35.337883 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:35.337845    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:35.524964 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:35.524931    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:39.934721 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:39.934688    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:40.090788 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:40.090753    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:40.709299 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:40.709264    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:40.988445 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:40.988354    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:41.751117 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:20:41.751080    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:20:46.345965 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:46.345929    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:46.551388 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:46.551358    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:49.250519 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:49.250487    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:49.727452 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:49.727377    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:50.022526 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:50.022496    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq"]
Nov 02 23:20:50.123640 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:50.123611    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt"]
Nov 02 23:20:50.389743 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:50.389670    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:50.547523 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:50.547487    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:51.713281 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:51.713244    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:55.022917 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.022879    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq" podUID="20c964fe-e6c9-4a32-bea1-e9e29845a78e" containerName="payload-processing" containerID="cri-o://570ec042b3631c6ce8206dfcc3f4c769922c159be3f688c66fbbb96ef2ea9ad2" gracePeriod=25
Nov 02 23:20:55.124148 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.124108    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt" podUID="d562d136-00fe-4689-8072-05b919e15697" containerName="payload-pre-processing" containerID="cri-o://3179f7f05d9352b52d634663d70d7e909fa456433bfeb3192d032b93508c0b37" gracePeriod=25
Nov 02 23:20:55.307958 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.307931    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq"
Nov 02 23:20:55.413381 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.413357    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt"
Nov 02 23:20:55.414022 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.413999    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:20:55.417405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.417382    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20c964fe-e6c9-4a32-bea1-e9e29845a78e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c964fe-e6c9-4a32-bea1-e9e29845a78e-plugins-config-volume\") pod \"20c964fe-e6c9-4a32-bea1-e9e29845a78e\" (UID: \"20c964fe-e6c9-4a32-bea1-e9e29845a78e\") "
Nov 02 23:20:55.417496 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.417477    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c964fe-e6c9-4a32-bea1-e9e29845a78e-kube-api-access-rth9z\" (UniqueName: \"kubernetes.io/projected/20c964fe-e6c9-4a32-bea1-e9e29845a78e-kube-api-access-rth9z\") pod \"20c964fe-e6c9-4a32-bea1-e9e29845a78e\" (UID: \"20c964fe-e6c9-4a32-bea1-e9e29845a78e\") "
Nov 02 23:20:55.417715 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.417693    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c964fe-e6c9-4a32-bea1-e9e29845a78e-plugins-config-volume" pod "20c964fe-e6c9-4a32-bea1-e9e29845a78e" (UID: "20c964fe-e6c9-4a32-bea1-e9e29845a78e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:20:55.417944 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.417926    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c964fe-e6c9-4a32-bea1-e9e29845a78e-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:20:55.420037 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.419972    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c964fe-e6c9-4a32-bea1-e9e29845a78e-kube-api-access-rth9z" pod "20c964fe-e6c9-4a32-bea1-e9e29845a78e" (UID: "20c964fe-e6c9-4a32-bea1-e9e29845a78e"). InnerVolumeSpecName "kube-api-access-rth9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:20:55.518652 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.518613    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d562d136-00fe-4689-8072-05b919e15697-kube-api-access-vdh5c\" (UniqueName: \"kubernetes.io/projected/d562d136-00fe-4689-8072-05b919e15697-kube-api-access-vdh5c\") pod \"d562d136-00fe-4689-8072-05b919e15697\" (UID: \"d562d136-00fe-4689-8072-05b919e15697\") "
Nov 02 23:20:55.518837 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.518708    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d562d136-00fe-4689-8072-05b919e15697-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d562d136-00fe-4689-8072-05b919e15697-plugins-config-volume\") pod \"d562d136-00fe-4689-8072-05b919e15697\" (UID: \"d562d136-00fe-4689-8072-05b919e15697\") "
Nov 02 23:20:55.519099 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.519065    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d562d136-00fe-4689-8072-05b919e15697-plugins-config-volume" pod "d562d136-00fe-4689-8072-05b919e15697" (UID: "d562d136-00fe-4689-8072-05b919e15697"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:20:55.519294 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.519280    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rth9z\" (UniqueName: \"kubernetes.io/projected/20c964fe-e6c9-4a32-bea1-e9e29845a78e-kube-api-access-rth9z\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:20:55.519363 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.519296    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d562d136-00fe-4689-8072-05b919e15697-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:20:55.520676 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.520658    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d562d136-00fe-4689-8072-05b919e15697-kube-api-access-vdh5c" pod "d562d136-00fe-4689-8072-05b919e15697" (UID: "d562d136-00fe-4689-8072-05b919e15697"). InnerVolumeSpecName "kube-api-access-vdh5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:20:55.605383 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.605301    4294 generic.go:356] "Generic (PLEG): container finished" podID="20c964fe-e6c9-4a32-bea1-e9e29845a78e" containerID="570ec042b3631c6ce8206dfcc3f4c769922c159be3f688c66fbbb96ef2ea9ad2" exitCode=0
Nov 02 23:20:55.605383 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.605344    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq"
Nov 02 23:20:55.605592 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.605375    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq" event={"ID":"20c964fe-e6c9-4a32-bea1-e9e29845a78e","Type":"ContainerDied","Data":"570ec042b3631c6ce8206dfcc3f4c769922c159be3f688c66fbbb96ef2ea9ad2"}
Nov 02 23:20:55.605592 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.605422    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq" event={"ID":"20c964fe-e6c9-4a32-bea1-e9e29845a78e","Type":"ContainerDied","Data":"611549435cdd72fc18b7c1f7b3e89b3f588baf76aa8289c4ab534f9d37b7eaec"}
Nov 02 23:20:55.605592 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.605451    4294 scope.go:122] "RemoveContainer" containerID="570ec042b3631c6ce8206dfcc3f4c769922c159be3f688c66fbbb96ef2ea9ad2"
Nov 02 23:20:55.610922 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.610898    4294 generic.go:356] "Generic (PLEG): container finished" podID="d562d136-00fe-4689-8072-05b919e15697" containerID="3179f7f05d9352b52d634663d70d7e909fa456433bfeb3192d032b93508c0b37" exitCode=0
Nov 02 23:20:55.611078 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.610947    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt"
Nov 02 23:20:55.611078 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.611009    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt" event={"ID":"d562d136-00fe-4689-8072-05b919e15697","Type":"ContainerDied","Data":"3179f7f05d9352b52d634663d70d7e909fa456433bfeb3192d032b93508c0b37"}
Nov 02 23:20:55.611078 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.611050    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt" event={"ID":"d562d136-00fe-4689-8072-05b919e15697","Type":"ContainerDied","Data":"857fed60902e477ad26b029321d0744b5716b188ba2d9cae27c48271f3334edc"}
Nov 02 23:20:55.620205 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.620184    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdh5c\" (UniqueName: \"kubernetes.io/projected/d562d136-00fe-4689-8072-05b919e15697-kube-api-access-vdh5c\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:20:55.620859 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.620838    4294 scope.go:122] "RemoveContainer" containerID="570ec042b3631c6ce8206dfcc3f4c769922c159be3f688c66fbbb96ef2ea9ad2"
Nov 02 23:20:55.621169 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:20:55.621152    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"570ec042b3631c6ce8206dfcc3f4c769922c159be3f688c66fbbb96ef2ea9ad2\": container with ID starting with 570ec042b3631c6ce8206dfcc3f4c769922c159be3f688c66fbbb96ef2ea9ad2 not found: ID does not exist" containerID="570ec042b3631c6ce8206dfcc3f4c769922c159be3f688c66fbbb96ef2ea9ad2"
Nov 02 23:20:55.621221 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.621176    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"570ec042b3631c6ce8206dfcc3f4c769922c159be3f688c66fbbb96ef2ea9ad2"} err="failed to get container status \"570ec042b3631c6ce8206dfcc3f4c769922c159be3f688c66fbbb96ef2ea9ad2\": rpc error: code = NotFound desc = could not find container \"570ec042b3631c6ce8206dfcc3f4c769922c159be3f688c66fbbb96ef2ea9ad2\": container with ID starting with 570ec042b3631c6ce8206dfcc3f4c769922c159be3f688c66fbbb96ef2ea9ad2 not found: ID does not exist"
Nov 02 23:20:55.621221 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.621193    4294 scope.go:122] "RemoveContainer" containerID="3179f7f05d9352b52d634663d70d7e909fa456433bfeb3192d032b93508c0b37"
Nov 02 23:20:55.629857 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.629842    4294 scope.go:122] "RemoveContainer" containerID="3179f7f05d9352b52d634663d70d7e909fa456433bfeb3192d032b93508c0b37"
Nov 02 23:20:55.630140 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:20:55.630122    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3179f7f05d9352b52d634663d70d7e909fa456433bfeb3192d032b93508c0b37\": container with ID starting with 3179f7f05d9352b52d634663d70d7e909fa456433bfeb3192d032b93508c0b37 not found: ID does not exist" containerID="3179f7f05d9352b52d634663d70d7e909fa456433bfeb3192d032b93508c0b37"
Nov 02 23:20:55.630215 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.630146    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3179f7f05d9352b52d634663d70d7e909fa456433bfeb3192d032b93508c0b37"} err="failed to get container status \"3179f7f05d9352b52d634663d70d7e909fa456433bfeb3192d032b93508c0b37\": rpc error: code = NotFound desc = could not find container \"3179f7f05d9352b52d634663d70d7e909fa456433bfeb3192d032b93508c0b37\": container with ID starting with 3179f7f05d9352b52d634663d70d7e909fa456433bfeb3192d032b93508c0b37 not found: ID does not exist"
Nov 02 23:20:55.763829 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.763799    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq"]
Nov 02 23:20:55.775130 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.775095    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e953257b-6f6fdc99f5-kdvfq"]
Nov 02 23:20:55.874661 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.874588    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt"]
Nov 02 23:20:55.879879 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:55.879854    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e953257b-6756fc6855-vhwvt"]
Nov 02 23:20:56.548230 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:56.548199    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20c964fe-e6c9-4a32-bea1-e9e29845a78e" path="/var/lib/kubelet/pods/20c964fe-e6c9-4a32-bea1-e9e29845a78e/volumes"
Nov 02 23:20:56.548608 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:56.548600    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d562d136-00fe-4689-8072-05b919e15697" path="/var/lib/kubelet/pods/d562d136-00fe-4689-8072-05b919e15697/volumes"
Nov 02 23:20:57.131013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:20:57.130965    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:02.154779 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:02.154741    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:02.316867 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:02.316837    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:02.482600 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:02.482520    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:02.645423 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:02.645390    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:07.885093 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:07.885055    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:08.043117 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:08.043082    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:11.750889 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:21:11.750852    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:21:12.777864 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:12.777833    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:17.784040 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:17.784004    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:22.717160 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:22.717130    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:24.453337 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:24.453279    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:27.991144 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:27.991116    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:28.204746 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:28.204715    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:29.817859 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:29.817817    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:31.172503 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:31.172440    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:31.335402 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:31.335373    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:32.880131 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:32.880094    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:36.011628 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:36.011579    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:36.968180 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:36.968146    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2"]
Nov 02 23:21:37.246596 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:37.246563    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:37.412750 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:37.412716    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:39.497130 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:39.497086    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:41.460857 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:41.460827    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88"]
Nov 02 23:21:41.562623 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:41.562590    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm"]
Nov 02 23:21:41.687590 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:41.687562    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:41.772385 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:21:41.772347    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:21:41.878036 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:41.876651    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:41.970837 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:41.970788    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2" podUID="c12aac19-4212-4954-bc46-10688e21eb09" containerName="payload-pre-processing" containerID="cri-o://402eee30ee17c4bb7ab0d20b941fc86de4b5f9c86ee7c8d0b243952836998063" gracePeriod=25
Nov 02 23:21:42.043517 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:42.043350    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:42.215350 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:42.215321    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:42.247390 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:42.247364    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2"
Nov 02 23:21:42.335190 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:42.335115    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c12aac19-4212-4954-bc46-10688e21eb09-kube-api-access-ct4g6\" (UniqueName: \"kubernetes.io/projected/c12aac19-4212-4954-bc46-10688e21eb09-kube-api-access-ct4g6\") pod \"c12aac19-4212-4954-bc46-10688e21eb09\" (UID: \"c12aac19-4212-4954-bc46-10688e21eb09\") "
Nov 02 23:21:42.335350 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:42.335268    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c12aac19-4212-4954-bc46-10688e21eb09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c12aac19-4212-4954-bc46-10688e21eb09-plugins-config-volume\") pod \"c12aac19-4212-4954-bc46-10688e21eb09\" (UID: \"c12aac19-4212-4954-bc46-10688e21eb09\") "
Nov 02 23:21:42.335663 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:42.335642    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12aac19-4212-4954-bc46-10688e21eb09-plugins-config-volume" pod "c12aac19-4212-4954-bc46-10688e21eb09" (UID: "c12aac19-4212-4954-bc46-10688e21eb09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:21:42.335773 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:42.335759    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c12aac19-4212-4954-bc46-10688e21eb09-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:21:42.337289 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:42.337268    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12aac19-4212-4954-bc46-10688e21eb09-kube-api-access-ct4g6" pod "c12aac19-4212-4954-bc46-10688e21eb09" (UID: "c12aac19-4212-4954-bc46-10688e21eb09"). InnerVolumeSpecName "kube-api-access-ct4g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:21:42.437567 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:42.437535    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct4g6\" (UniqueName: \"kubernetes.io/projected/c12aac19-4212-4954-bc46-10688e21eb09-kube-api-access-ct4g6\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:21:42.918458 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:42.918422    4294 generic.go:356] "Generic (PLEG): container finished" podID="c12aac19-4212-4954-bc46-10688e21eb09" containerID="402eee30ee17c4bb7ab0d20b941fc86de4b5f9c86ee7c8d0b243952836998063" exitCode=0
Nov 02 23:21:42.919072 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:42.918491    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2"
Nov 02 23:21:42.920230 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:42.918509    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2" event={"ID":"c12aac19-4212-4954-bc46-10688e21eb09","Type":"ContainerDied","Data":"402eee30ee17c4bb7ab0d20b941fc86de4b5f9c86ee7c8d0b243952836998063"}
Nov 02 23:21:42.920230 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:42.919700    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2" event={"ID":"c12aac19-4212-4954-bc46-10688e21eb09","Type":"ContainerDied","Data":"158f75f8ebe87042d37a064c5f2ea892656cafbe705a04c2a3b893d21da946ca"}
Nov 02 23:21:42.920230 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:42.919740    4294 scope.go:122] "RemoveContainer" containerID="402eee30ee17c4bb7ab0d20b941fc86de4b5f9c86ee7c8d0b243952836998063"
Nov 02 23:21:42.936991 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:42.936942    4294 scope.go:122] "RemoveContainer" containerID="402eee30ee17c4bb7ab0d20b941fc86de4b5f9c86ee7c8d0b243952836998063"
Nov 02 23:21:42.937389 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:21:42.937359    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"402eee30ee17c4bb7ab0d20b941fc86de4b5f9c86ee7c8d0b243952836998063\": container with ID starting with 402eee30ee17c4bb7ab0d20b941fc86de4b5f9c86ee7c8d0b243952836998063 not found: ID does not exist" containerID="402eee30ee17c4bb7ab0d20b941fc86de4b5f9c86ee7c8d0b243952836998063"
Nov 02 23:21:42.937468 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:42.937403    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"402eee30ee17c4bb7ab0d20b941fc86de4b5f9c86ee7c8d0b243952836998063"} err="failed to get container status \"402eee30ee17c4bb7ab0d20b941fc86de4b5f9c86ee7c8d0b243952836998063\": rpc error: code = NotFound desc = could not find container \"402eee30ee17c4bb7ab0d20b941fc86de4b5f9c86ee7c8d0b243952836998063\": container with ID starting with 402eee30ee17c4bb7ab0d20b941fc86de4b5f9c86ee7c8d0b243952836998063 not found: ID does not exist"
Nov 02 23:21:43.079146 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:43.079072    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2"]
Nov 02 23:21:43.128603 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:43.128560    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-68e067-66674fb667-f7bk2"]
Nov 02 23:21:43.899652 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:43.899620    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:44.067643 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:44.067609    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:44.548714 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:44.548682    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c12aac19-4212-4954-bc46-10688e21eb09" path="/var/lib/kubelet/pods/c12aac19-4212-4954-bc46-10688e21eb09/volumes"
Nov 02 23:21:44.724749 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:44.724721    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:45.814811 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:45.814777    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:46.373552 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.373516    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:46.461884 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.461842    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88" podUID="87085fd9-8b23-4f20-8819-562321e3b11e" containerName="payload-processing" containerID="cri-o://669ddfbba1a0cf388e3300122568d3004f617fb26bd7ff50a441246ca2594588" gracePeriod=25
Nov 02 23:21:46.563450 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.563405    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm" podUID="443dc393-32b0-4ca3-be71-7d6f77747162" containerName="payload-pre-processing" containerID="cri-o://2713a7c62b6f441430d90c10a1d39f70c34a83a12981e9a2e3138ff6d0ba80b0" gracePeriod=25
Nov 02 23:21:46.725835 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.725808    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88"
Nov 02 23:21:46.802863 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.802831    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87085fd9-8b23-4f20-8819-562321e3b11e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87085fd9-8b23-4f20-8819-562321e3b11e-plugins-config-volume\") pod \"87085fd9-8b23-4f20-8819-562321e3b11e\" (UID: \"87085fd9-8b23-4f20-8819-562321e3b11e\") "
Nov 02 23:21:46.803083 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.802963    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87085fd9-8b23-4f20-8819-562321e3b11e-kube-api-access-x4qbp\" (UniqueName: \"kubernetes.io/projected/87085fd9-8b23-4f20-8819-562321e3b11e-kube-api-access-x4qbp\") pod \"87085fd9-8b23-4f20-8819-562321e3b11e\" (UID: \"87085fd9-8b23-4f20-8819-562321e3b11e\") "
Nov 02 23:21:46.803353 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.803324    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87085fd9-8b23-4f20-8819-562321e3b11e-plugins-config-volume" pod "87085fd9-8b23-4f20-8819-562321e3b11e" (UID: "87085fd9-8b23-4f20-8819-562321e3b11e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:21:46.803792 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.803766    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87085fd9-8b23-4f20-8819-562321e3b11e-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:21:46.805806 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.805770    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87085fd9-8b23-4f20-8819-562321e3b11e-kube-api-access-x4qbp" pod "87085fd9-8b23-4f20-8819-562321e3b11e" (UID: "87085fd9-8b23-4f20-8819-562321e3b11e"). InnerVolumeSpecName "kube-api-access-x4qbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:21:46.828498 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.828477    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm"
Nov 02 23:21:46.904674 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.904598    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/443dc393-32b0-4ca3-be71-7d6f77747162-kube-api-access-zbhqv\" (UniqueName: \"kubernetes.io/projected/443dc393-32b0-4ca3-be71-7d6f77747162-kube-api-access-zbhqv\") pod \"443dc393-32b0-4ca3-be71-7d6f77747162\" (UID: \"443dc393-32b0-4ca3-be71-7d6f77747162\") "
Nov 02 23:21:46.904819 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.904730    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/443dc393-32b0-4ca3-be71-7d6f77747162-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/443dc393-32b0-4ca3-be71-7d6f77747162-plugins-config-volume\") pod \"443dc393-32b0-4ca3-be71-7d6f77747162\" (UID: \"443dc393-32b0-4ca3-be71-7d6f77747162\") "
Nov 02 23:21:46.905119 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.905096    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/443dc393-32b0-4ca3-be71-7d6f77747162-plugins-config-volume" pod "443dc393-32b0-4ca3-be71-7d6f77747162" (UID: "443dc393-32b0-4ca3-be71-7d6f77747162"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:21:46.905230 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.905181    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/443dc393-32b0-4ca3-be71-7d6f77747162-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:21:46.905230 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.905195    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4qbp\" (UniqueName: \"kubernetes.io/projected/87085fd9-8b23-4f20-8819-562321e3b11e-kube-api-access-x4qbp\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:21:46.906639 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.906619    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/443dc393-32b0-4ca3-be71-7d6f77747162-kube-api-access-zbhqv" pod "443dc393-32b0-4ca3-be71-7d6f77747162" (UID: "443dc393-32b0-4ca3-be71-7d6f77747162"). InnerVolumeSpecName "kube-api-access-zbhqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:21:46.952192 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.952158    4294 generic.go:356] "Generic (PLEG): container finished" podID="443dc393-32b0-4ca3-be71-7d6f77747162" containerID="2713a7c62b6f441430d90c10a1d39f70c34a83a12981e9a2e3138ff6d0ba80b0" exitCode=0
Nov 02 23:21:46.952348 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.952206    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm"
Nov 02 23:21:46.952348 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.952236    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm" event={"ID":"443dc393-32b0-4ca3-be71-7d6f77747162","Type":"ContainerDied","Data":"2713a7c62b6f441430d90c10a1d39f70c34a83a12981e9a2e3138ff6d0ba80b0"}
Nov 02 23:21:46.952348 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.952291    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm" event={"ID":"443dc393-32b0-4ca3-be71-7d6f77747162","Type":"ContainerDied","Data":"ff0537bd2993c59d2d8fe3058295f9aa40d80e3cca8d546be6c9facbcec08c6d"}
Nov 02 23:21:46.952348 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.952316    4294 scope.go:122] "RemoveContainer" containerID="2713a7c62b6f441430d90c10a1d39f70c34a83a12981e9a2e3138ff6d0ba80b0"
Nov 02 23:21:46.953856 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.953833    4294 generic.go:356] "Generic (PLEG): container finished" podID="87085fd9-8b23-4f20-8819-562321e3b11e" containerID="669ddfbba1a0cf388e3300122568d3004f617fb26bd7ff50a441246ca2594588" exitCode=0
Nov 02 23:21:46.953999 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.953878    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88"
Nov 02 23:21:46.953999 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.953904    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88" event={"ID":"87085fd9-8b23-4f20-8819-562321e3b11e","Type":"ContainerDied","Data":"669ddfbba1a0cf388e3300122568d3004f617fb26bd7ff50a441246ca2594588"}
Nov 02 23:21:46.953999 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.953937    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88" event={"ID":"87085fd9-8b23-4f20-8819-562321e3b11e","Type":"ContainerDied","Data":"8a2e1b454042832c6428124c4164dc9293c23513ad5d15482bbfeb3732610c38"}
Nov 02 23:21:46.964273 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.964252    4294 scope.go:122] "RemoveContainer" containerID="2713a7c62b6f441430d90c10a1d39f70c34a83a12981e9a2e3138ff6d0ba80b0"
Nov 02 23:21:46.964562 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:21:46.964535    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2713a7c62b6f441430d90c10a1d39f70c34a83a12981e9a2e3138ff6d0ba80b0\": container with ID starting with 2713a7c62b6f441430d90c10a1d39f70c34a83a12981e9a2e3138ff6d0ba80b0 not found: ID does not exist" containerID="2713a7c62b6f441430d90c10a1d39f70c34a83a12981e9a2e3138ff6d0ba80b0"
Nov 02 23:21:46.964713 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.964573    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2713a7c62b6f441430d90c10a1d39f70c34a83a12981e9a2e3138ff6d0ba80b0"} err="failed to get container status \"2713a7c62b6f441430d90c10a1d39f70c34a83a12981e9a2e3138ff6d0ba80b0\": rpc error: code = NotFound desc = could not find container \"2713a7c62b6f441430d90c10a1d39f70c34a83a12981e9a2e3138ff6d0ba80b0\": container with ID starting with 2713a7c62b6f441430d90c10a1d39f70c34a83a12981e9a2e3138ff6d0ba80b0 not found: ID does not exist"
Nov 02 23:21:46.964713 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.964599    4294 scope.go:122] "RemoveContainer" containerID="669ddfbba1a0cf388e3300122568d3004f617fb26bd7ff50a441246ca2594588"
Nov 02 23:21:46.974089 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.974068    4294 scope.go:122] "RemoveContainer" containerID="669ddfbba1a0cf388e3300122568d3004f617fb26bd7ff50a441246ca2594588"
Nov 02 23:21:46.974366 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:21:46.974350    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"669ddfbba1a0cf388e3300122568d3004f617fb26bd7ff50a441246ca2594588\": container with ID starting with 669ddfbba1a0cf388e3300122568d3004f617fb26bd7ff50a441246ca2594588 not found: ID does not exist" containerID="669ddfbba1a0cf388e3300122568d3004f617fb26bd7ff50a441246ca2594588"
Nov 02 23:21:46.974412 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:46.974373    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"669ddfbba1a0cf388e3300122568d3004f617fb26bd7ff50a441246ca2594588"} err="failed to get container status \"669ddfbba1a0cf388e3300122568d3004f617fb26bd7ff50a441246ca2594588\": rpc error: code = NotFound desc = could not find container \"669ddfbba1a0cf388e3300122568d3004f617fb26bd7ff50a441246ca2594588\": container with ID starting with 669ddfbba1a0cf388e3300122568d3004f617fb26bd7ff50a441246ca2594588 not found: ID does not exist"
Nov 02 23:21:47.006944 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:47.006916    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbhqv\" (UniqueName: \"kubernetes.io/projected/443dc393-32b0-4ca3-be71-7d6f77747162-kube-api-access-zbhqv\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:21:47.113348 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:47.113318    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm"]
Nov 02 23:21:47.116576 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:47.116550    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a69f65-764b57f4cf-rswrm"]
Nov 02 23:21:47.221175 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:47.221100    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88"]
Nov 02 23:21:47.226572 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:47.226544    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a69f65-845d56cdc-ssb88"]
Nov 02 23:21:47.508383 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:47.508336    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:48.096488 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:48.096194    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:48.548461 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:48.548431    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="443dc393-32b0-4ca3-be71-7d6f77747162" path="/var/lib/kubelet/pods/443dc393-32b0-4ca3-be71-7d6f77747162/volumes"
Nov 02 23:21:48.548864 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:48.548850    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87085fd9-8b23-4f20-8819-562321e3b11e" path="/var/lib/kubelet/pods/87085fd9-8b23-4f20-8819-562321e3b11e/volumes"
Nov 02 23:21:48.694945 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:48.694702    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:48.862686 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:48.862529    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:49.376317 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:49.375231    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:51.122137 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.122105    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv"]
Nov 02 23:21:51.122841 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.122820    4294 cpu_manager.go:414] "containerMap: removing container" podUID="d562d136-00fe-4689-8072-05b919e15697" containerName="payload-pre-processing"
Nov 02 23:21:51.122938 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.122843    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d562d136-00fe-4689-8072-05b919e15697" containerName="payload-pre-processing"
Nov 02 23:21:51.122938 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.122868    4294 cpu_manager.go:414] "containerMap: removing container" podUID="87085fd9-8b23-4f20-8819-562321e3b11e" containerName="payload-processing"
Nov 02 23:21:51.122938 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.122875    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87085fd9-8b23-4f20-8819-562321e3b11e" containerName="payload-processing"
Nov 02 23:21:51.122938 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.122906    4294 cpu_manager.go:414] "containerMap: removing container" podUID="c12aac19-4212-4954-bc46-10688e21eb09" containerName="payload-pre-processing"
Nov 02 23:21:51.122938 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.122913    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c12aac19-4212-4954-bc46-10688e21eb09" containerName="payload-pre-processing"
Nov 02 23:21:51.122938 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.122935    4294 cpu_manager.go:414] "containerMap: removing container" podUID="443dc393-32b0-4ca3-be71-7d6f77747162" containerName="payload-pre-processing"
Nov 02 23:21:51.123274 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.122943    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="443dc393-32b0-4ca3-be71-7d6f77747162" containerName="payload-pre-processing"
Nov 02 23:21:51.123274 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.122993    4294 cpu_manager.go:414] "containerMap: removing container" podUID="20c964fe-e6c9-4a32-bea1-e9e29845a78e" containerName="payload-processing"
Nov 02 23:21:51.123274 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.123001    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c964fe-e6c9-4a32-bea1-e9e29845a78e" containerName="payload-processing"
Nov 02 23:21:51.123274 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.123140    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="443dc393-32b0-4ca3-be71-7d6f77747162" containerName="payload-pre-processing"
Nov 02 23:21:51.123274 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.123159    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c964fe-e6c9-4a32-bea1-e9e29845a78e" containerName="payload-processing"
Nov 02 23:21:51.123274 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.123172    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d562d136-00fe-4689-8072-05b919e15697" containerName="payload-pre-processing"
Nov 02 23:21:51.123274 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.123183    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="87085fd9-8b23-4f20-8819-562321e3b11e" containerName="payload-processing"
Nov 02 23:21:51.123274 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.123196    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c12aac19-4212-4954-bc46-10688e21eb09" containerName="payload-pre-processing"
Nov 02 23:21:51.128282 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.128261    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv"
Nov 02 23:21:51.163758 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.163728    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14520749-ad0f-4518-a370-e5559f3fd565-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv\" (UID: \"14520749-ad0f-4518-a370-e5559f3fd565\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv"
Nov 02 23:21:51.163898 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.163878    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbgtc\" (UniqueName: \"kubernetes.io/projected/14520749-ad0f-4518-a370-e5559f3fd565-kube-api-access-wbgtc\") pod \"payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv\" (UID: \"14520749-ad0f-4518-a370-e5559f3fd565\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv"
Nov 02 23:21:51.181359 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.181336    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-027689\""
Nov 02 23:21:51.181468 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.181373    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-027689-dockercfg-pfhhd\""
Nov 02 23:21:51.199799 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.199770    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv"]
Nov 02 23:21:51.265615 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.265585    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14520749-ad0f-4518-a370-e5559f3fd565-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv\" (UID: \"14520749-ad0f-4518-a370-e5559f3fd565\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv"
Nov 02 23:21:51.265755 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.265736    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbgtc\" (UniqueName: \"kubernetes.io/projected/14520749-ad0f-4518-a370-e5559f3fd565-kube-api-access-wbgtc\") pod \"payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv\" (UID: \"14520749-ad0f-4518-a370-e5559f3fd565\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv"
Nov 02 23:21:51.266231 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.266214    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14520749-ad0f-4518-a370-e5559f3fd565-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv\" (UID: \"14520749-ad0f-4518-a370-e5559f3fd565\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv"
Nov 02 23:21:51.320060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.320033    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbgtc\" (UniqueName: \"kubernetes.io/projected/14520749-ad0f-4518-a370-e5559f3fd565-kube-api-access-wbgtc\") pod \"payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv\" (UID: \"14520749-ad0f-4518-a370-e5559f3fd565\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv"
Nov 02 23:21:51.441084 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.441008    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv"
Nov 02 23:21:51.812306 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.812275    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv"]
Nov 02 23:21:51.812706 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:21:51.812684    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14520749_ad0f_4518_a370_e5559f3fd565.slice/crio-26e1fc2acfd123119e1dcb747158e77f447a129da07e7c74d6872f91a775e042 WatchSource:0}: Error finding container 26e1fc2acfd123119e1dcb747158e77f447a129da07e7c74d6872f91a775e042: Status 404 returned error can't find the container with id 26e1fc2acfd123119e1dcb747158e77f447a129da07e7c74d6872f91a775e042
Nov 02 23:21:51.814988 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.814960    4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 23:21:51.986931 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:51.986894    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv" event={"ID":"14520749-ad0f-4518-a370-e5559f3fd565","Type":"ContainerStarted","Data":"26e1fc2acfd123119e1dcb747158e77f447a129da07e7c74d6872f91a775e042"}
Nov 02 23:21:52.265919 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:52.265889    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:52.427945 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:52.427906    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:52.601335 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:52.601254    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:55.007352 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:55.007319    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv" event={"ID":"14520749-ad0f-4518-a370-e5559f3fd565","Type":"ContainerStarted","Data":"8f023eee4dad8c5d4ee3433932f152fdd6dd42332511508091baa04aea34ea7f"}
Nov 02 23:21:55.007874 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:55.007424    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv"
Nov 02 23:21:55.117586 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:55.117535    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv" podStartSLOduration=1.984618866 podStartE2EDuration="4.11752087s" totalImagesPullingTime="2.132902004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:21:55.115356818 +0000 UTC m=+1754.683711605" watchObservedRunningTime="2026-11-02 23:21:55.11752087 +0000 UTC m=+1754.685875656"
Nov 02 23:21:55.573846 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:55.573814    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:55.747782 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:55.747750    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:55.959210 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:55.959132    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk"]
Nov 02 23:21:55.963205 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:55.963188    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk"
Nov 02 23:21:56.019807 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.019771    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk"]
Nov 02 23:21:56.028687 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.028651    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hlmc\" (UniqueName: \"kubernetes.io/projected/c2b82c9b-91dd-44e6-a474-4a80b62e916f-kube-api-access-9hlmc\") pod \"payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk\" (UID: \"c2b82c9b-91dd-44e6-a474-4a80b62e916f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk"
Nov 02 23:21:56.029015 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.028992    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b82c9b-91dd-44e6-a474-4a80b62e916f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk\" (UID: \"c2b82c9b-91dd-44e6-a474-4a80b62e916f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk"
Nov 02 23:21:56.061291 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.061260    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6a894c-dockercfg-6xnwm\""
Nov 02 23:21:56.061613 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.061593    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6a894c\""
Nov 02 23:21:56.120737 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.120704    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq"]
Nov 02 23:21:56.125720 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.125695    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq"
Nov 02 23:21:56.130476 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.130454    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hlmc\" (UniqueName: \"kubernetes.io/projected/c2b82c9b-91dd-44e6-a474-4a80b62e916f-kube-api-access-9hlmc\") pod \"payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk\" (UID: \"c2b82c9b-91dd-44e6-a474-4a80b62e916f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk"
Nov 02 23:21:56.130637 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.130612    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b82c9b-91dd-44e6-a474-4a80b62e916f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk\" (UID: \"c2b82c9b-91dd-44e6-a474-4a80b62e916f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk"
Nov 02 23:21:56.131334 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.131316    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b82c9b-91dd-44e6-a474-4a80b62e916f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk\" (UID: \"c2b82c9b-91dd-44e6-a474-4a80b62e916f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk"
Nov 02 23:21:56.172964 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.172928    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq"]
Nov 02 23:21:56.185233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.185204    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hlmc\" (UniqueName: \"kubernetes.io/projected/c2b82c9b-91dd-44e6-a474-4a80b62e916f-kube-api-access-9hlmc\") pod \"payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk\" (UID: \"c2b82c9b-91dd-44e6-a474-4a80b62e916f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk"
Nov 02 23:21:56.232258 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.232128    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbjs8\" (UniqueName: \"kubernetes.io/projected/5c71bd6a-003a-4245-a685-6b19be14c826-kube-api-access-dbjs8\") pod \"payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq\" (UID: \"5c71bd6a-003a-4245-a685-6b19be14c826\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq"
Nov 02 23:21:56.233083 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.233048    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c71bd6a-003a-4245-a685-6b19be14c826-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq\" (UID: \"5c71bd6a-003a-4245-a685-6b19be14c826\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq"
Nov 02 23:21:56.276433 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.276405    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk"
Nov 02 23:21:56.334305 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.334266    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbjs8\" (UniqueName: \"kubernetes.io/projected/5c71bd6a-003a-4245-a685-6b19be14c826-kube-api-access-dbjs8\") pod \"payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq\" (UID: \"5c71bd6a-003a-4245-a685-6b19be14c826\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq"
Nov 02 23:21:56.334434 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.334420    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c71bd6a-003a-4245-a685-6b19be14c826-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq\" (UID: \"5c71bd6a-003a-4245-a685-6b19be14c826\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq"
Nov 02 23:21:56.335194 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.335170    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c71bd6a-003a-4245-a685-6b19be14c826-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq\" (UID: \"5c71bd6a-003a-4245-a685-6b19be14c826\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq"
Nov 02 23:21:56.388937 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.388904    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbjs8\" (UniqueName: \"kubernetes.io/projected/5c71bd6a-003a-4245-a685-6b19be14c826-kube-api-access-dbjs8\") pod \"payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq\" (UID: \"5c71bd6a-003a-4245-a685-6b19be14c826\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq"
Nov 02 23:21:56.441063 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.441028    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq"
Nov 02 23:21:56.647778 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:21:56.647739    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b82c9b_91dd_44e6_a474_4a80b62e916f.slice/crio-2630aade0398744a2ddaedb83ae8a235061a640ad631b4ee4c55f32768b8a61c WatchSource:0}: Error finding container 2630aade0398744a2ddaedb83ae8a235061a640ad631b4ee4c55f32768b8a61c: Status 404 returned error can't find the container with id 2630aade0398744a2ddaedb83ae8a235061a640ad631b4ee4c55f32768b8a61c
Nov 02 23:21:56.647927 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.647904    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk"]
Nov 02 23:21:56.808469 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:21:56.808433    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c71bd6a_003a_4245_a685_6b19be14c826.slice/crio-79657fdb6e08a827f467d9c76570409a9f930b7be073ada9ab388778112c473b WatchSource:0}: Error finding container 79657fdb6e08a827f467d9c76570409a9f930b7be073ada9ab388778112c473b: Status 404 returned error can't find the container with id 79657fdb6e08a827f467d9c76570409a9f930b7be073ada9ab388778112c473b
Nov 02 23:21:56.809014 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.808974    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq"]
Nov 02 23:21:56.965035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:56.965004    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:21:57.031295 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:57.030945    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk" event={"ID":"c2b82c9b-91dd-44e6-a474-4a80b62e916f","Type":"ContainerStarted","Data":"68af350dce90df9ab175004bf438136708635abda9d3b33a0b35b344bdf63a2a"}
Nov 02 23:21:57.031295 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:57.031085    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk" event={"ID":"c2b82c9b-91dd-44e6-a474-4a80b62e916f","Type":"ContainerStarted","Data":"2630aade0398744a2ddaedb83ae8a235061a640ad631b4ee4c55f32768b8a61c"}
Nov 02 23:21:57.032954 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:57.032892    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk"
Nov 02 23:21:57.035725 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:57.035698    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq" event={"ID":"5c71bd6a-003a-4245-a685-6b19be14c826","Type":"ContainerStarted","Data":"1724442d717d7893cae28e15814e0316a11e3337ad78d804fa7ded0a738d8cf0"}
Nov 02 23:21:57.035850 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:57.035750    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq" event={"ID":"5c71bd6a-003a-4245-a685-6b19be14c826","Type":"ContainerStarted","Data":"79657fdb6e08a827f467d9c76570409a9f930b7be073ada9ab388778112c473b"}
Nov 02 23:21:57.038992 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:57.038922    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq"
Nov 02 23:21:57.144898 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:57.144794    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk" podStartSLOduration=2.144779352 podStartE2EDuration="2.144779352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:21:57.142492799 +0000 UTC m=+1756.710847587" watchObservedRunningTime="2026-11-02 23:21:57.144779352 +0000 UTC m=+1756.713134137"
Nov 02 23:21:57.252060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:57.252011    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq" podStartSLOduration=1.251998182 podStartE2EDuration="1.251998182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:21:57.250291945 +0000 UTC m=+1756.818646731" watchObservedRunningTime="2026-11-02 23:21:57.251998182 +0000 UTC m=+1756.820352983"
Nov 02 23:21:58.921496 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:21:58.921466    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:04.103647 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:04.103603    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:05.570093 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:05.570059    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:05.745533 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:05.745499    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:06.018721 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:06.018691    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv"
Nov 02 23:22:07.292120 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:07.292077    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:07.692927 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:07.692843    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:08.475466 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:08.475415    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:09.051644 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:09.051610    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq"
Nov 02 23:22:09.051853 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:09.051717    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk"
Nov 02 23:22:09.346820 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:09.346736    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:10.063158 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:10.063125    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:11.750900 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:22:11.750867    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:22:12.285771 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:12.285736    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:14.505063 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:14.505006    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:14.822345 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:14.822271    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:15.297278 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:15.297244    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:15.480112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:15.480076    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:15.937292 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:15.937261    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:16.165718 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:16.165670    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:16.480769 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:16.480735    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:19.416510 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:19.416472    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:19.625146 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:19.625094    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:20.351803 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:20.351754    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:20.514534 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:20.514484    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:25.691739 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:25.691702    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:29.376437 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:29.376404    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:29.900203 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:29.900155    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:31.513459 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:31.513402    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:32.707272 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:32.707228    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:36.923788 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:36.923749    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:38.670901 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:38.670863    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:38.943944 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:38.943860    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:39.145278 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:39.145237    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"]
Nov 02 23:22:39.145462 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:39.145441    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth" podUID="07652258-9b4f-41e6-bcaf-54ee4c68049d" containerName="istio-proxy" containerID="cri-o://c78e8f83a0b84131e9a708d270ea545b8d6850c3c264742b640606970ef2696c" gracePeriod=30
Nov 02 23:22:39.151877 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:39.151848    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"]
Nov 02 23:22:39.152120 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:39.152094    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2" podUID="d080b437-c462-4b4d-8f36-575abce2a2f1" containerName="istio-proxy" containerID="cri-o://33b21398550ab7003b3331e18cd1b21fa388264a87e07228dbbdcc2954374fa5" gracePeriod=30
Nov 02 23:22:39.167579 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:39.167548    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:39.228696 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:39.228616    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"]
Nov 02 23:22:39.228848 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:39.228813    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn" podUID="d6735e93-95bc-4514-8739-31725bf4fd5d" containerName="istio-proxy" containerID="cri-o://37c4fcd5bec4b7567570b4d4653ce7d9d1ce25162863df8b420a33e3c0b22771" gracePeriod=30
Nov 02 23:22:40.027422 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:40.027387    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:40.623476 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:40.623431    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:41.006193 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:41.006161    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:41.174276 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:41.174243    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:41.751015 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:22:41.750951    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:22:44.369350 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.369319    4294 generic.go:356] "Generic (PLEG): container finished" podID="d080b437-c462-4b4d-8f36-575abce2a2f1" containerID="33b21398550ab7003b3331e18cd1b21fa388264a87e07228dbbdcc2954374fa5" exitCode=0
Nov 02 23:22:44.369845 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.369414    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2" event={"ID":"d080b437-c462-4b4d-8f36-575abce2a2f1","Type":"ContainerDied","Data":"33b21398550ab7003b3331e18cd1b21fa388264a87e07228dbbdcc2954374fa5"}
Nov 02 23:22:44.371493 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.371467    4294 generic.go:356] "Generic (PLEG): container finished" podID="d6735e93-95bc-4514-8739-31725bf4fd5d" containerID="37c4fcd5bec4b7567570b4d4653ce7d9d1ce25162863df8b420a33e3c0b22771" exitCode=0
Nov 02 23:22:44.371617 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.371548    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn" event={"ID":"d6735e93-95bc-4514-8739-31725bf4fd5d","Type":"ContainerDied","Data":"37c4fcd5bec4b7567570b4d4653ce7d9d1ce25162863df8b420a33e3c0b22771"}
Nov 02 23:22:44.371617 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.371597    4294 scope.go:122] "RemoveContainer" containerID="94bddf21811a60a483efc68980b67ed91f0d4b84103c92251a212498aad9ba00"
Nov 02 23:22:44.374369 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.374350    4294 generic.go:356] "Generic (PLEG): container finished" podID="07652258-9b4f-41e6-bcaf-54ee4c68049d" containerID="c78e8f83a0b84131e9a708d270ea545b8d6850c3c264742b640606970ef2696c" exitCode=0
Nov 02 23:22:44.374467 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.374389    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth" event={"ID":"07652258-9b4f-41e6-bcaf-54ee4c68049d","Type":"ContainerDied","Data":"c78e8f83a0b84131e9a708d270ea545b8d6850c3c264742b640606970ef2696c"}
Nov 02 23:22:44.458621 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.458585    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:22:44.487833 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.487804    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:22:44.552831 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.546499    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:22:44.590069 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.590035    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-data\") pod \"07652258-9b4f-41e6-bcaf-54ee4c68049d\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") "
Nov 02 23:22:44.590269 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.590120    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-token\" (UniqueName: \"kubernetes.io/projected/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-token\") pod \"07652258-9b4f-41e6-bcaf-54ee4c68049d\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") "
Nov 02 23:22:44.590269 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.590171    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-workload-certs\") pod \"d080b437-c462-4b4d-8f36-575abce2a2f1\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") "
Nov 02 23:22:44.590269 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.590218    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-envoy\") pod \"d080b437-c462-4b4d-8f36-575abce2a2f1\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") "
Nov 02 23:22:44.590269 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.590267    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-workload-socket\") pod \"d080b437-c462-4b4d-8f36-575abce2a2f1\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") "
Nov 02 23:22:44.590550 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.590295    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-podinfo\") pod \"07652258-9b4f-41e6-bcaf-54ee4c68049d\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") "
Nov 02 23:22:44.590550 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.590349    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-envoy\") pod \"07652258-9b4f-41e6-bcaf-54ee4c68049d\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") "
Nov 02 23:22:44.590550 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.590352    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-data" pod "07652258-9b4f-41e6-bcaf-54ee4c68049d" (UID: "07652258-9b4f-41e6-bcaf-54ee4c68049d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:22:44.590550 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.590390    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07652258-9b4f-41e6-bcaf-54ee4c68049d-kube-api-access-cmlmt\" (UniqueName: \"kubernetes.io/projected/07652258-9b4f-41e6-bcaf-54ee4c68049d-kube-api-access-cmlmt\") pod \"07652258-9b4f-41e6-bcaf-54ee4c68049d\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") "
Nov 02 23:22:44.590550 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.590438    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-workload-certs\") pod \"07652258-9b4f-41e6-bcaf-54ee4c68049d\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") "
Nov 02 23:22:44.590550 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.590480    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d080b437-c462-4b4d-8f36-575abce2a2f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d080b437-c462-4b4d-8f36-575abce2a2f1-istiod-ca-cert\") pod \"d080b437-c462-4b4d-8f36-575abce2a2f1\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") "
Nov 02 23:22:44.590550 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.590499    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-workload-socket\") pod \"07652258-9b4f-41e6-bcaf-54ee4c68049d\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") "
Nov 02 23:22:44.590550 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.590527    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-credential-socket\") pod \"07652258-9b4f-41e6-bcaf-54ee4c68049d\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") "
Nov 02 23:22:44.590953 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.590568    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-credential-socket\") pod \"d080b437-c462-4b4d-8f36-575abce2a2f1\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") "
Nov 02 23:22:44.590953 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.590603    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-token\" (UniqueName: \"kubernetes.io/projected/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-token\") pod \"d080b437-c462-4b4d-8f36-575abce2a2f1\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") "
Nov 02 23:22:44.590953 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.590646    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07652258-9b4f-41e6-bcaf-54ee4c68049d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07652258-9b4f-41e6-bcaf-54ee4c68049d-istiod-ca-cert\") pod \"07652258-9b4f-41e6-bcaf-54ee4c68049d\" (UID: \"07652258-9b4f-41e6-bcaf-54ee4c68049d\") "
Nov 02 23:22:44.590953 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.590739    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-workload-certs" pod "d080b437-c462-4b4d-8f36-575abce2a2f1" (UID: "d080b437-c462-4b4d-8f36-575abce2a2f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:22:44.590953 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.590943    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-workload-socket" pod "07652258-9b4f-41e6-bcaf-54ee4c68049d" (UID: "07652258-9b4f-41e6-bcaf-54ee4c68049d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:22:44.591233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.590956    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-workload-socket" pod "d080b437-c462-4b4d-8f36-575abce2a2f1" (UID: "d080b437-c462-4b4d-8f36-575abce2a2f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:22:44.591233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.591059    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-podinfo\") pod \"d080b437-c462-4b4d-8f36-575abce2a2f1\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") "
Nov 02 23:22:44.591233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.591096    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d080b437-c462-4b4d-8f36-575abce2a2f1-kube-api-access-98lfv\" (UniqueName: \"kubernetes.io/projected/d080b437-c462-4b4d-8f36-575abce2a2f1-kube-api-access-98lfv\") pod \"d080b437-c462-4b4d-8f36-575abce2a2f1\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") "
Nov 02 23:22:44.591233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.591138    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-data\") pod \"d080b437-c462-4b4d-8f36-575abce2a2f1\" (UID: \"d080b437-c462-4b4d-8f36-575abce2a2f1\") "
Nov 02 23:22:44.591679 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.591654    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.591782 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.591685    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.591782 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.591712    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.591782 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.591722    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.592843 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.592815    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-credential-socket" pod "d080b437-c462-4b4d-8f36-575abce2a2f1" (UID: "d080b437-c462-4b4d-8f36-575abce2a2f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:22:44.593879 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.593550    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d080b437-c462-4b4d-8f36-575abce2a2f1-istiod-ca-cert" pod "d080b437-c462-4b4d-8f36-575abce2a2f1" (UID: "d080b437-c462-4b4d-8f36-575abce2a2f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:22:44.593879 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.593557    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07652258-9b4f-41e6-bcaf-54ee4c68049d-istiod-ca-cert" pod "07652258-9b4f-41e6-bcaf-54ee4c68049d" (UID: "07652258-9b4f-41e6-bcaf-54ee4c68049d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:22:44.594068 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.593920    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-podinfo" pod "07652258-9b4f-41e6-bcaf-54ee4c68049d" (UID: "07652258-9b4f-41e6-bcaf-54ee4c68049d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:22:44.594068 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.594002    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-credential-socket" pod "07652258-9b4f-41e6-bcaf-54ee4c68049d" (UID: "07652258-9b4f-41e6-bcaf-54ee4c68049d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:22:44.594318 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.594191    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-data" pod "d080b437-c462-4b4d-8f36-575abce2a2f1" (UID: "d080b437-c462-4b4d-8f36-575abce2a2f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:22:44.594820 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.594790    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-workload-certs" pod "07652258-9b4f-41e6-bcaf-54ee4c68049d" (UID: "07652258-9b4f-41e6-bcaf-54ee4c68049d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:22:44.595152 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.595120    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-envoy" pod "d080b437-c462-4b4d-8f36-575abce2a2f1" (UID: "d080b437-c462-4b4d-8f36-575abce2a2f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:22:44.595426 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.595345    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-token" pod "07652258-9b4f-41e6-bcaf-54ee4c68049d" (UID: "07652258-9b4f-41e6-bcaf-54ee4c68049d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:22:44.595726 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.595561    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07652258-9b4f-41e6-bcaf-54ee4c68049d-kube-api-access-cmlmt" pod "07652258-9b4f-41e6-bcaf-54ee4c68049d" (UID: "07652258-9b4f-41e6-bcaf-54ee4c68049d"). InnerVolumeSpecName "kube-api-access-cmlmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:22:44.596060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.596036    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-podinfo" pod "d080b437-c462-4b4d-8f36-575abce2a2f1" (UID: "d080b437-c462-4b4d-8f36-575abce2a2f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:22:44.596964 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.596939    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d080b437-c462-4b4d-8f36-575abce2a2f1-kube-api-access-98lfv" pod "d080b437-c462-4b4d-8f36-575abce2a2f1" (UID: "d080b437-c462-4b4d-8f36-575abce2a2f1"). InnerVolumeSpecName "kube-api-access-98lfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:22:44.597629 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.597607    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-envoy" pod "07652258-9b4f-41e6-bcaf-54ee4c68049d" (UID: "07652258-9b4f-41e6-bcaf-54ee4c68049d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:22:44.598135 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.598114    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-token" pod "d080b437-c462-4b4d-8f36-575abce2a2f1" (UID: "d080b437-c462-4b4d-8f36-575abce2a2f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:22:44.693049 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.692938    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6735e93-95bc-4514-8739-31725bf4fd5d-kube-api-access-s2kzk\" (UniqueName: \"kubernetes.io/projected/d6735e93-95bc-4514-8739-31725bf4fd5d-kube-api-access-s2kzk\") pod \"d6735e93-95bc-4514-8739-31725bf4fd5d\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") "
Nov 02 23:22:44.693049 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.693017    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-workload-certs\") pod \"d6735e93-95bc-4514-8739-31725bf4fd5d\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") "
Nov 02 23:22:44.693049 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.693047    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-data\") pod \"d6735e93-95bc-4514-8739-31725bf4fd5d\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") "
Nov 02 23:22:44.693435 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.693110    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6735e93-95bc-4514-8739-31725bf4fd5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6735e93-95bc-4514-8739-31725bf4fd5d-istiod-ca-cert\") pod \"d6735e93-95bc-4514-8739-31725bf4fd5d\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") "
Nov 02 23:22:44.693435 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.693130    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-token\" (UniqueName: \"kubernetes.io/projected/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-token\") pod \"d6735e93-95bc-4514-8739-31725bf4fd5d\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") "
Nov 02 23:22:44.693435 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.693155    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-envoy\") pod \"d6735e93-95bc-4514-8739-31725bf4fd5d\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") "
Nov 02 23:22:44.693435 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.693194    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-credential-socket\") pod \"d6735e93-95bc-4514-8739-31725bf4fd5d\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") "
Nov 02 23:22:44.693435 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.693362    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-workload-certs" pod "d6735e93-95bc-4514-8739-31725bf4fd5d" (UID: "d6735e93-95bc-4514-8739-31725bf4fd5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:22:44.693697 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.693470    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-data" pod "d6735e93-95bc-4514-8739-31725bf4fd5d" (UID: "d6735e93-95bc-4514-8739-31725bf4fd5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:22:44.693697 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.693509    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-credential-socket" pod "d6735e93-95bc-4514-8739-31725bf4fd5d" (UID: "d6735e93-95bc-4514-8739-31725bf4fd5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:22:44.693697 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.693522    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6735e93-95bc-4514-8739-31725bf4fd5d-istiod-ca-cert" pod "d6735e93-95bc-4514-8739-31725bf4fd5d" (UID: "d6735e93-95bc-4514-8739-31725bf4fd5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:22:44.693697 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.693640    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-workload-socket\") pod \"d6735e93-95bc-4514-8739-31725bf4fd5d\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") "
Nov 02 23:22:44.693697 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.693677    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-podinfo\") pod \"d6735e93-95bc-4514-8739-31725bf4fd5d\" (UID: \"d6735e93-95bc-4514-8739-31725bf4fd5d\") "
Nov 02 23:22:44.693945 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.693873    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-workload-socket" pod "d6735e93-95bc-4514-8739-31725bf4fd5d" (UID: "d6735e93-95bc-4514-8739-31725bf4fd5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:22:44.694546 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.694520    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.694694 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.694552    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.694694 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.694569    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.694694 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.694584    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.694694 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.694602    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmlmt\" (UniqueName: \"kubernetes.io/projected/07652258-9b4f-41e6-bcaf-54ee4c68049d-kube-api-access-cmlmt\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.694694 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.694620    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6735e93-95bc-4514-8739-31725bf4fd5d-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.694694 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.694636    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.694694 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.694651    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.694694 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.694668    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d080b437-c462-4b4d-8f36-575abce2a2f1-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.694694 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.694683    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07652258-9b4f-41e6-bcaf-54ee4c68049d-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.694694 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.694698    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.695264 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.694711    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.695264 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.694722    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07652258-9b4f-41e6-bcaf-54ee4c68049d-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.695264 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.694731    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.695264 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.694740    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98lfv\" (UniqueName: \"kubernetes.io/projected/d080b437-c462-4b4d-8f36-575abce2a2f1-kube-api-access-98lfv\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.695264 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.694748    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.695264 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.694756    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.695264 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.694765    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07652258-9b4f-41e6-bcaf-54ee4c68049d-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.695264 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.694774    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d080b437-c462-4b4d-8f36-575abce2a2f1-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.695730 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.695708    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-podinfo" pod "d6735e93-95bc-4514-8739-31725bf4fd5d" (UID: "d6735e93-95bc-4514-8739-31725bf4fd5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:22:44.695859 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.695845    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-envoy" pod "d6735e93-95bc-4514-8739-31725bf4fd5d" (UID: "d6735e93-95bc-4514-8739-31725bf4fd5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:22:44.695916 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.695859    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-token" pod "d6735e93-95bc-4514-8739-31725bf4fd5d" (UID: "d6735e93-95bc-4514-8739-31725bf4fd5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:22:44.696249 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.696235    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6735e93-95bc-4514-8739-31725bf4fd5d-kube-api-access-s2kzk" pod "d6735e93-95bc-4514-8739-31725bf4fd5d" (UID: "d6735e93-95bc-4514-8739-31725bf4fd5d"). InnerVolumeSpecName "kube-api-access-s2kzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:22:44.803836 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.803798    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.803836 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.803836    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2kzk\" (UniqueName: \"kubernetes.io/projected/d6735e93-95bc-4514-8739-31725bf4fd5d-kube-api-access-s2kzk\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.804126 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.803852    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.804126 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.803866    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6735e93-95bc-4514-8739-31725bf4fd5d-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:22:44.925662 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:44.925628    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:45.108894 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:45.108852    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:45.276321 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:45.276285    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:45.382121 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:45.382034    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"
Nov 02 23:22:45.382121 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:45.382022    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2" event={"ID":"d080b437-c462-4b4d-8f36-575abce2a2f1","Type":"ContainerDied","Data":"51125c66bf9ca5a4539a83d6398db212d69b2ec2bd2cf77adcc4aa2d85d7d8b8"}
Nov 02 23:22:45.382630 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:45.382170    4294 scope.go:122] "RemoveContainer" containerID="33b21398550ab7003b3331e18cd1b21fa388264a87e07228dbbdcc2954374fa5"
Nov 02 23:22:45.383670 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:45.383651    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"
Nov 02 23:22:45.384290 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:45.384054    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn" event={"ID":"d6735e93-95bc-4514-8739-31725bf4fd5d","Type":"ContainerDied","Data":"088ecc0d04dd622af85f81521df68588d22ac8a8ef1fafa2ec6d439f738a519d"}
Nov 02 23:22:45.392068 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:45.391963    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth" event={"ID":"07652258-9b4f-41e6-bcaf-54ee4c68049d","Type":"ContainerDied","Data":"c81047c5e25f3a545ae67f4bf4dac2a80c81ff1cc89b6e068200429082dc9227"}
Nov 02 23:22:45.392194 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:45.392097    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"
Nov 02 23:22:45.399003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:45.398854    4294 scope.go:122] "RemoveContainer" containerID="37c4fcd5bec4b7567570b4d4653ce7d9d1ce25162863df8b420a33e3c0b22771"
Nov 02 23:22:45.408811 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:45.408792    4294 scope.go:122] "RemoveContainer" containerID="c78e8f83a0b84131e9a708d270ea545b8d6850c3c264742b640606970ef2696c"
Nov 02 23:22:45.594729 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:45.594686    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"]
Nov 02 23:22:45.621792 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:45.621759    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-rzzvn"]
Nov 02 23:22:45.704018 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:45.703920    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"]
Nov 02 23:22:45.706869 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:45.706846    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-cmdth"]
Nov 02 23:22:45.810845 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:45.810808    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"]
Nov 02 23:22:45.814903 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:45.814877    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e953257b-openshift-default-77d7b959b5-fmts2"]
Nov 02 23:22:46.202435 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:46.200343    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:46.385303 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:46.385265    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:46.548515 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:46.548485    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07652258-9b4f-41e6-bcaf-54ee4c68049d" path="/var/lib/kubelet/pods/07652258-9b4f-41e6-bcaf-54ee4c68049d/volumes"
Nov 02 23:22:46.549012 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:46.548996    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d080b437-c462-4b4d-8f36-575abce2a2f1" path="/var/lib/kubelet/pods/d080b437-c462-4b4d-8f36-575abce2a2f1/volumes"
Nov 02 23:22:46.549451 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:46.549438    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6735e93-95bc-4514-8739-31725bf4fd5d" path="/var/lib/kubelet/pods/d6735e93-95bc-4514-8739-31725bf4fd5d/volumes"
Nov 02 23:22:46.661574 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:46.661545    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:46.837107 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:46.837025    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:49.558865 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:49.558835    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:50.068532 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.068497    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"]
Nov 02 23:22:50.069079 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.069062    4294 cpu_manager.go:414] "containerMap: removing container" podUID="d6735e93-95bc-4514-8739-31725bf4fd5d" containerName="istio-proxy"
Nov 02 23:22:50.069079 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.069077    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6735e93-95bc-4514-8739-31725bf4fd5d" containerName="istio-proxy"
Nov 02 23:22:50.069271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.069092    4294 cpu_manager.go:414] "containerMap: removing container" podUID="d080b437-c462-4b4d-8f36-575abce2a2f1" containerName="istio-proxy"
Nov 02 23:22:50.069271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.069098    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d080b437-c462-4b4d-8f36-575abce2a2f1" containerName="istio-proxy"
Nov 02 23:22:50.069271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.069120    4294 cpu_manager.go:414] "containerMap: removing container" podUID="07652258-9b4f-41e6-bcaf-54ee4c68049d" containerName="istio-proxy"
Nov 02 23:22:50.069271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.069127    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07652258-9b4f-41e6-bcaf-54ee4c68049d" containerName="istio-proxy"
Nov 02 23:22:50.069271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.069202    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6735e93-95bc-4514-8739-31725bf4fd5d" containerName="istio-proxy"
Nov 02 23:22:50.069271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.069211    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d080b437-c462-4b4d-8f36-575abce2a2f1" containerName="istio-proxy"
Nov 02 23:22:50.069271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.069217    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6735e93-95bc-4514-8739-31725bf4fd5d" containerName="istio-proxy"
Nov 02 23:22:50.069271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.069225    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="07652258-9b4f-41e6-bcaf-54ee4c68049d" containerName="istio-proxy"
Nov 02 23:22:50.075035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.074891    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.179239 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.179208    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9f5790-openshift-default-dockercfg-djmzg\""
Nov 02 23:22:50.184023 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.182151    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:50.184023 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.183539    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"]
Nov 02 23:22:50.278902 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.278870    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-workload-socket\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.279132 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.278933    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-data\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.279132 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.279049    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-token\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.279132 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.279117    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-credential-socket\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.279310 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.279144    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwq6l\" (UniqueName: \"kubernetes.io/projected/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-kube-api-access-lwq6l\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.279310 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.279189    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-podinfo\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.279431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.279387    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-envoy\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.279498 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.279444    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istiod-ca-cert\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.279498 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.279479    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-workload-certs\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.380548 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.380467    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-envoy\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.380548 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.380515    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istiod-ca-cert\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.380548 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.380539    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-workload-certs\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.380786 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.380591    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-workload-socket\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.380786 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.380658    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-data\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.380786 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.380696    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-token\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.380786 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.380761    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-credential-socket\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.380975 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.380792    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwq6l\" (UniqueName: \"kubernetes.io/projected/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-kube-api-access-lwq6l\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.380975 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.380836    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-podinfo\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.380975 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.380915    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-workload-certs\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.381244 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.381217    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-data\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.381358 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.381312    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-credential-socket\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.381358 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.381325    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istiod-ca-cert\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.381436 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.381333    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-workload-socket\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.382871 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.382842    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-envoy\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.383299 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.383283    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-podinfo\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.438682 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.438647    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-token\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.438836 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.438657    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwq6l\" (UniqueName: \"kubernetes.io/projected/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-kube-api-access-lwq6l\") pod \"e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") " pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.491863 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.491822    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:50.697192 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.697112    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:50.747143 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.747109    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:50.922627 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:50.922597    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:51.122956 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:51.122925    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"]
Nov 02 23:22:51.123076 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:22:51.122969    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9daec5e_f7ee_4d62_be3f_3d6a3ef1b9e2.slice/crio-b5dbc4804aea02f92da70521bd0f42c443f44a313c1510567b5bfd8721496775 WatchSource:0}: Error finding container b5dbc4804aea02f92da70521bd0f42c443f44a313c1510567b5bfd8721496775: Status 404 returned error can't find the container with id b5dbc4804aea02f92da70521bd0f42c443f44a313c1510567b5bfd8721496775
Nov 02 23:22:51.125418 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:51.125387    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:22:51.125481 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:51.125459    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:22:51.125515 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:51.125485    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:22:51.433378 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:51.433307    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb" event={"ID":"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2","Type":"ContainerStarted","Data":"39e32cfa3308105a603e7508711f246bc3206b5a8d787e290bf5efe8d27f176e"}
Nov 02 23:22:51.433378 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:51.433356    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb" event={"ID":"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2","Type":"ContainerStarted","Data":"b5dbc4804aea02f92da70521bd0f42c443f44a313c1510567b5bfd8721496775"}
Nov 02 23:22:51.547861 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:51.547794    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb" podStartSLOduration=1.5477762940000002 podStartE2EDuration="1.547776294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:22:51.546901956 +0000 UTC m=+1811.115256743" watchObservedRunningTime="2026-11-02 23:22:51.547776294 +0000 UTC m=+1811.116131081"
Nov 02 23:22:51.637033 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:51.637001    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:51.698078 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:51.697991    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:52.703326 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:52.703296    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:53.445328 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:53.445294    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:53.446399 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:53.446374    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:22:54.379729 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:54.379696    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:54.555673 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:54.555626    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:54.725428 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:54.725349    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:56.975212 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:56.975181    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:57.532586 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:57.532555    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:57.702304 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:57.702266    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:22:58.059950 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:58.059918    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t"]
Nov 02 23:22:58.060777 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:58.060754    4294 cpu_manager.go:414] "containerMap: removing container" podUID="d6735e93-95bc-4514-8739-31725bf4fd5d" containerName="istio-proxy"
Nov 02 23:22:58.061514 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:58.061483    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6735e93-95bc-4514-8739-31725bf4fd5d" containerName="istio-proxy"
Nov 02 23:22:58.065764 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:58.065741    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t"
Nov 02 23:22:58.116192 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:58.116160    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t"]
Nov 02 23:22:58.158363 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:58.158329    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9f5790\""
Nov 02 23:22:58.159881 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:58.159861    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9f5790-dockercfg-wxnn6\""
Nov 02 23:22:58.181609 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:58.181580    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbmnn\" (UniqueName: \"kubernetes.io/projected/6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e-kube-api-access-fbmnn\") pod \"payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t\" (UID: \"6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t"
Nov 02 23:22:58.181760 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:58.181619    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t\" (UID: \"6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t"
Nov 02 23:22:58.283342 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:58.283297    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbmnn\" (UniqueName: \"kubernetes.io/projected/6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e-kube-api-access-fbmnn\") pod \"payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t\" (UID: \"6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t"
Nov 02 23:22:58.283342 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:58.283344    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t\" (UID: \"6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t"
Nov 02 23:22:58.284030 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:58.283972    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t\" (UID: \"6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t"
Nov 02 23:22:58.340930 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:58.340853    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbmnn\" (UniqueName: \"kubernetes.io/projected/6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e-kube-api-access-fbmnn\") pod \"payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t\" (UID: \"6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t"
Nov 02 23:22:58.386881 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:58.386848    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t"
Nov 02 23:22:58.753813 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:22:58.753784    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2e14f3_dcc6_4e7d_9e41_1ba6e01b972e.slice/crio-6bee9efa294879b3102aae57e7e3e8ff6971bf18e7e37e6c5b0463789903ec57 WatchSource:0}: Error finding container 6bee9efa294879b3102aae57e7e3e8ff6971bf18e7e37e6c5b0463789903ec57: Status 404 returned error can't find the container with id 6bee9efa294879b3102aae57e7e3e8ff6971bf18e7e37e6c5b0463789903ec57
Nov 02 23:22:58.753953 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:58.753816    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t"]
Nov 02 23:22:59.483263 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:59.483227    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t" event={"ID":"6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e","Type":"ContainerStarted","Data":"cc8ead1cec3c14cde7cc1ca02614f9c5222e4468edee064b9387cb0acde50b64"}
Nov 02 23:22:59.483673 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:59.483281    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t" event={"ID":"6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e","Type":"ContainerStarted","Data":"6bee9efa294879b3102aae57e7e3e8ff6971bf18e7e37e6c5b0463789903ec57"}
Nov 02 23:22:59.483673 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:59.483437    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t"
Nov 02 23:22:59.592204 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:22:59.592152    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t" podStartSLOduration=1.592136923 podStartE2EDuration="1.592136923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:22:59.591145612 +0000 UTC m=+1819.159500402" watchObservedRunningTime="2026-11-02 23:22:59.592136923 +0000 UTC m=+1819.160491711"
Nov 02 23:23:02.298754 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:02.298719    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:02.410477 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:02.410439    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"]
Nov 02 23:23:02.410751 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:02.410704    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5" podUID="33529202-6346-4b2a-b3ad-59dc8b8fa59a" containerName="istio-proxy" containerID="cri-o://8defac415c8184cb5b308d823b9deee784c3e10551f74937907bf56636458473" gracePeriod=30
Nov 02 23:23:06.236094 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:06.236062    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 23:23:06.236507 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:06.236113    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5" podUID="33529202-6346-4b2a-b3ad-59dc8b8fa59a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 23:23:07.080472 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.080439    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:07.535349 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.535322    4294 generic.go:356] "Generic (PLEG): container finished" podID="33529202-6346-4b2a-b3ad-59dc8b8fa59a" containerID="8defac415c8184cb5b308d823b9deee784c3e10551f74937907bf56636458473" exitCode=0
Nov 02 23:23:07.535696 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.535409    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5" event={"ID":"33529202-6346-4b2a-b3ad-59dc8b8fa59a","Type":"ContainerDied","Data":"8defac415c8184cb5b308d823b9deee784c3e10551f74937907bf56636458473"}
Nov 02 23:23:07.669385 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.669363    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:23:07.802107 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.802063    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-workload-certs\") pod \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") "
Nov 02 23:23:07.802314 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.802126    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-credential-socket\") pod \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") "
Nov 02 23:23:07.802314 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.802193    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33529202-6346-4b2a-b3ad-59dc8b8fa59a-kube-api-access-sljjp\" (UniqueName: \"kubernetes.io/projected/33529202-6346-4b2a-b3ad-59dc8b8fa59a-kube-api-access-sljjp\") pod \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") "
Nov 02 23:23:07.802314 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.802253    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-workload-socket\") pod \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") "
Nov 02 23:23:07.802488 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.802359    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-data\") pod \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") "
Nov 02 23:23:07.802488 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.802366    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-workload-certs" pod "33529202-6346-4b2a-b3ad-59dc8b8fa59a" (UID: "33529202-6346-4b2a-b3ad-59dc8b8fa59a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:23:07.802488 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.802405    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-token\" (UniqueName: \"kubernetes.io/projected/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-token\") pod \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") "
Nov 02 23:23:07.802488 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.802446    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-credential-socket" pod "33529202-6346-4b2a-b3ad-59dc8b8fa59a" (UID: "33529202-6346-4b2a-b3ad-59dc8b8fa59a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:23:07.802488 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.802458    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-podinfo\") pod \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") "
Nov 02 23:23:07.802786 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.802525    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-envoy\") pod \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") "
Nov 02 23:23:07.802786 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.802609    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-workload-socket" pod "33529202-6346-4b2a-b3ad-59dc8b8fa59a" (UID: "33529202-6346-4b2a-b3ad-59dc8b8fa59a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:23:07.802786 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.802621    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-data" pod "33529202-6346-4b2a-b3ad-59dc8b8fa59a" (UID: "33529202-6346-4b2a-b3ad-59dc8b8fa59a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:23:07.802786 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.802623    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istiod-ca-cert\") pod \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\" (UID: \"33529202-6346-4b2a-b3ad-59dc8b8fa59a\") "
Nov 02 23:23:07.803024 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.802916    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istiod-ca-cert" pod "33529202-6346-4b2a-b3ad-59dc8b8fa59a" (UID: "33529202-6346-4b2a-b3ad-59dc8b8fa59a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:23:07.803275 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.803251    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:23:07.803369 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.803280    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:23:07.803369 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.803295    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:23:07.803369 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.803308    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:23:07.803369 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.803322    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:23:07.804690 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.804666    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-token" pod "33529202-6346-4b2a-b3ad-59dc8b8fa59a" (UID: "33529202-6346-4b2a-b3ad-59dc8b8fa59a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:23:07.804896 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.804868    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33529202-6346-4b2a-b3ad-59dc8b8fa59a-kube-api-access-sljjp" pod "33529202-6346-4b2a-b3ad-59dc8b8fa59a" (UID: "33529202-6346-4b2a-b3ad-59dc8b8fa59a"). InnerVolumeSpecName "kube-api-access-sljjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:23:07.804896 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.804877    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-podinfo" pod "33529202-6346-4b2a-b3ad-59dc8b8fa59a" (UID: "33529202-6346-4b2a-b3ad-59dc8b8fa59a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:23:07.805074 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.804959    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-envoy" pod "33529202-6346-4b2a-b3ad-59dc8b8fa59a" (UID: "33529202-6346-4b2a-b3ad-59dc8b8fa59a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:23:07.904058 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.904025    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:23:07.904058 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.904051    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:23:07.904058 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.904061    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33529202-6346-4b2a-b3ad-59dc8b8fa59a-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:23:07.904290 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:07.904070    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sljjp\" (UniqueName: \"kubernetes.io/projected/33529202-6346-4b2a-b3ad-59dc8b8fa59a-kube-api-access-sljjp\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:23:08.542812 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:08.542782    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"
Nov 02 23:23:08.548860 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:08.548831    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5" event={"ID":"33529202-6346-4b2a-b3ad-59dc8b8fa59a","Type":"ContainerDied","Data":"efbc2b015a9535bfb8cbed9352d5a0f99b1fc0de6b4ce0b6d79f6abce833c8b1"}
Nov 02 23:23:08.549015 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:08.548882    4294 scope.go:122] "RemoveContainer" containerID="8defac415c8184cb5b308d823b9deee784c3e10551f74937907bf56636458473"
Nov 02 23:23:08.748888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:08.748858    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"]
Nov 02 23:23:08.795585 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:08.795505    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-gp7q5"]
Nov 02 23:23:08.811169 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:08.811134    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:09.013113 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:09.013035    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:09.169443 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:09.169343    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:09.798387 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:09.798329    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:09.957353 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:09.957288    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:10.492995 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:10.492958    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t"
Nov 02 23:23:10.548192 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:10.548163    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33529202-6346-4b2a-b3ad-59dc8b8fa59a" path="/var/lib/kubelet/pods/33529202-6346-4b2a-b3ad-59dc8b8fa59a/volumes"
Nov 02 23:23:11.363209 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:11.363179    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:11.782255 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:23:11.776314    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:23:11.840263 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:11.840232    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:12.725079 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:12.725044    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:13.604613 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:13.604577    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:13.917761 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:13.917650    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:14.372202 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:14.372166    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:17.161930 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:17.161900    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:17.354721 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:17.354688    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:17.539423 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:17.539391    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:18.729715 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:18.729687    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:19.788187 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:19.788150    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:20.344991 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:20.344947    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:20.526920 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:20.526890    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:20.917795 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:20.917763    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j"]
Nov 02 23:23:20.918633 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:20.918609    4294 cpu_manager.go:414] "containerMap: removing container" podUID="33529202-6346-4b2a-b3ad-59dc8b8fa59a" containerName="istio-proxy"
Nov 02 23:23:20.918740 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:20.918634    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33529202-6346-4b2a-b3ad-59dc8b8fa59a" containerName="istio-proxy"
Nov 02 23:23:20.918835 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:20.918821    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="33529202-6346-4b2a-b3ad-59dc8b8fa59a" containerName="istio-proxy"
Nov 02 23:23:20.923872 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:20.923845    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j"
Nov 02 23:23:20.989115 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:20.989081    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b473b0-75fd-4d11-8670-3ffd0abb90ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j\" (UID: \"46b473b0-75fd-4d11-8670-3ffd0abb90ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j"
Nov 02 23:23:20.989311 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:20.989148    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qltrl\" (UniqueName: \"kubernetes.io/projected/46b473b0-75fd-4d11-8670-3ffd0abb90ac-kube-api-access-qltrl\") pod \"payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j\" (UID: \"46b473b0-75fd-4d11-8670-3ffd0abb90ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j"
Nov 02 23:23:20.993853 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:20.993827    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j"]
Nov 02 23:23:21.028638 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:21.028612    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-981724\""
Nov 02 23:23:21.028811 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:21.028734    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-981724-dockercfg-7b5g5\""
Nov 02 23:23:21.090365 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:21.090335    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b473b0-75fd-4d11-8670-3ffd0abb90ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j\" (UID: \"46b473b0-75fd-4d11-8670-3ffd0abb90ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j"
Nov 02 23:23:21.090524 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:21.090379    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qltrl\" (UniqueName: \"kubernetes.io/projected/46b473b0-75fd-4d11-8670-3ffd0abb90ac-kube-api-access-qltrl\") pod \"payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j\" (UID: \"46b473b0-75fd-4d11-8670-3ffd0abb90ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j"
Nov 02 23:23:21.091024 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:21.091004    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b473b0-75fd-4d11-8670-3ffd0abb90ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j\" (UID: \"46b473b0-75fd-4d11-8670-3ffd0abb90ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j"
Nov 02 23:23:21.147494 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:21.147467    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qltrl\" (UniqueName: \"kubernetes.io/projected/46b473b0-75fd-4d11-8670-3ffd0abb90ac-kube-api-access-qltrl\") pod \"payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j\" (UID: \"46b473b0-75fd-4d11-8670-3ffd0abb90ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j"
Nov 02 23:23:21.238309 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:21.238236    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j"
Nov 02 23:23:21.623372 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:23:21.623345    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b473b0_75fd_4d11_8670_3ffd0abb90ac.slice/crio-323ed99d02d88cdc85c08547becbae5c365c03d7068c20b7f013ec0b7fcedb84 WatchSource:0}: Error finding container 323ed99d02d88cdc85c08547becbae5c365c03d7068c20b7f013ec0b7fcedb84: Status 404 returned error can't find the container with id 323ed99d02d88cdc85c08547becbae5c365c03d7068c20b7f013ec0b7fcedb84
Nov 02 23:23:21.623713 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:21.623690    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j"]
Nov 02 23:23:21.642851 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:21.642821    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j" event={"ID":"46b473b0-75fd-4d11-8670-3ffd0abb90ac","Type":"ContainerStarted","Data":"323ed99d02d88cdc85c08547becbae5c365c03d7068c20b7f013ec0b7fcedb84"}
Nov 02 23:23:22.584449 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:22.584413    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"]
Nov 02 23:23:22.584852 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:22.584610    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk" podUID="602de6ee-5748-44d1-b389-7124d80e521b" containerName="istio-proxy" containerID="cri-o://1d2490d1278ee6eb8ab4e430d731848635d5fb526d2d655ea46d6f87c65a6e66" gracePeriod=30
Nov 02 23:23:22.653610 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:22.653571    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j" event={"ID":"46b473b0-75fd-4d11-8670-3ffd0abb90ac","Type":"ContainerStarted","Data":"bd6dee7988c11ff5d07a381d985caec12a4972cb011bc49371a67fe40f3ba3b5"}
Nov 02 23:23:22.653788 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:22.653642    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j"
Nov 02 23:23:22.807389 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:22.807326    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j" podStartSLOduration=2.80731273 podStartE2EDuration="2.80731273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:23:22.805505761 +0000 UTC m=+1842.373860550" watchObservedRunningTime="2026-11-02 23:23:22.80731273 +0000 UTC m=+1842.375667859"
Nov 02 23:23:25.193810 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:25.193733    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:27.860759 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.860700    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:23:27.975037 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.975005    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/602de6ee-5748-44d1-b389-7124d80e521b-istio-token\" (UniqueName: \"kubernetes.io/projected/602de6ee-5748-44d1-b389-7124d80e521b-istio-token\") pod \"602de6ee-5748-44d1-b389-7124d80e521b\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") "
Nov 02 23:23:27.975197 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.975045    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-workload-socket\") pod \"602de6ee-5748-44d1-b389-7124d80e521b\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") "
Nov 02 23:23:27.975197 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.975077    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/602de6ee-5748-44d1-b389-7124d80e521b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/602de6ee-5748-44d1-b389-7124d80e521b-istiod-ca-cert\") pod \"602de6ee-5748-44d1-b389-7124d80e521b\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") "
Nov 02 23:23:27.975197 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.975176    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-credential-socket\") pod \"602de6ee-5748-44d1-b389-7124d80e521b\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") "
Nov 02 23:23:27.975329 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.975216    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-workload-certs\") pod \"602de6ee-5748-44d1-b389-7124d80e521b\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") "
Nov 02 23:23:27.975329 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.975238    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-istio-data\") pod \"602de6ee-5748-44d1-b389-7124d80e521b\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") "
Nov 02 23:23:27.975329 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.975301    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/602de6ee-5748-44d1-b389-7124d80e521b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/602de6ee-5748-44d1-b389-7124d80e521b-istio-podinfo\") pod \"602de6ee-5748-44d1-b389-7124d80e521b\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") "
Nov 02 23:23:27.975504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.975335    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-workload-socket" pod "602de6ee-5748-44d1-b389-7124d80e521b" (UID: "602de6ee-5748-44d1-b389-7124d80e521b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:23:27.975504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.975349    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/602de6ee-5748-44d1-b389-7124d80e521b-kube-api-access-tqxp5\" (UniqueName: \"kubernetes.io/projected/602de6ee-5748-44d1-b389-7124d80e521b-kube-api-access-tqxp5\") pod \"602de6ee-5748-44d1-b389-7124d80e521b\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") "
Nov 02 23:23:27.975504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.975389    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-istio-envoy\") pod \"602de6ee-5748-44d1-b389-7124d80e521b\" (UID: \"602de6ee-5748-44d1-b389-7124d80e521b\") "
Nov 02 23:23:27.975504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.975459    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-credential-socket" pod "602de6ee-5748-44d1-b389-7124d80e521b" (UID: "602de6ee-5748-44d1-b389-7124d80e521b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:23:27.975504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.975496    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-workload-certs" pod "602de6ee-5748-44d1-b389-7124d80e521b" (UID: "602de6ee-5748-44d1-b389-7124d80e521b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:23:27.975763 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.975604    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-istio-data" pod "602de6ee-5748-44d1-b389-7124d80e521b" (UID: "602de6ee-5748-44d1-b389-7124d80e521b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:23:27.975815 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.975783    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602de6ee-5748-44d1-b389-7124d80e521b-istiod-ca-cert" pod "602de6ee-5748-44d1-b389-7124d80e521b" (UID: "602de6ee-5748-44d1-b389-7124d80e521b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:23:27.975934 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.975914    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:23:27.976041 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.975940    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:23:27.976041 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.975954    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:23:27.976041 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.975969    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:23:27.976041 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.976008    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/602de6ee-5748-44d1-b389-7124d80e521b-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:23:27.977317 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.977290    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602de6ee-5748-44d1-b389-7124d80e521b-istio-token" pod "602de6ee-5748-44d1-b389-7124d80e521b" (UID: "602de6ee-5748-44d1-b389-7124d80e521b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:23:27.977469 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.977451    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/602de6ee-5748-44d1-b389-7124d80e521b-istio-podinfo" pod "602de6ee-5748-44d1-b389-7124d80e521b" (UID: "602de6ee-5748-44d1-b389-7124d80e521b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:23:27.977839 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.977817    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-istio-envoy" pod "602de6ee-5748-44d1-b389-7124d80e521b" (UID: "602de6ee-5748-44d1-b389-7124d80e521b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:23:27.977943 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:27.977844    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602de6ee-5748-44d1-b389-7124d80e521b-kube-api-access-tqxp5" pod "602de6ee-5748-44d1-b389-7124d80e521b" (UID: "602de6ee-5748-44d1-b389-7124d80e521b"). InnerVolumeSpecName "kube-api-access-tqxp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:23:28.077560 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:28.077526    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/602de6ee-5748-44d1-b389-7124d80e521b-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:23:28.077560 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:28.077554    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqxp5\" (UniqueName: \"kubernetes.io/projected/602de6ee-5748-44d1-b389-7124d80e521b-kube-api-access-tqxp5\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:23:28.077560 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:28.077566    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/602de6ee-5748-44d1-b389-7124d80e521b-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:23:28.077779 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:28.077575    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/602de6ee-5748-44d1-b389-7124d80e521b-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:23:28.690861 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:28.690825    4294 generic.go:356] "Generic (PLEG): container finished" podID="602de6ee-5748-44d1-b389-7124d80e521b" containerID="1d2490d1278ee6eb8ab4e430d731848635d5fb526d2d655ea46d6f87c65a6e66" exitCode=0
Nov 02 23:23:28.691063 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:28.690878    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"
Nov 02 23:23:28.691063 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:28.690910    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk" event={"ID":"602de6ee-5748-44d1-b389-7124d80e521b","Type":"ContainerDied","Data":"1d2490d1278ee6eb8ab4e430d731848635d5fb526d2d655ea46d6f87c65a6e66"}
Nov 02 23:23:28.691063 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:28.690969    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk" event={"ID":"602de6ee-5748-44d1-b389-7124d80e521b","Type":"ContainerDied","Data":"951b646c00ecf5463663e9a3d5c38608d50a82db0e50cf799e6c71e10723376f"}
Nov 02 23:23:28.691063 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:28.691013    4294 scope.go:122] "RemoveContainer" containerID="1d2490d1278ee6eb8ab4e430d731848635d5fb526d2d655ea46d6f87c65a6e66"
Nov 02 23:23:28.700815 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:28.700799    4294 scope.go:122] "RemoveContainer" containerID="1d2490d1278ee6eb8ab4e430d731848635d5fb526d2d655ea46d6f87c65a6e66"
Nov 02 23:23:28.701106 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:23:28.701087    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d2490d1278ee6eb8ab4e430d731848635d5fb526d2d655ea46d6f87c65a6e66\": container with ID starting with 1d2490d1278ee6eb8ab4e430d731848635d5fb526d2d655ea46d6f87c65a6e66 not found: ID does not exist" containerID="1d2490d1278ee6eb8ab4e430d731848635d5fb526d2d655ea46d6f87c65a6e66"
Nov 02 23:23:28.701162 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:28.701112    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d2490d1278ee6eb8ab4e430d731848635d5fb526d2d655ea46d6f87c65a6e66"} err="failed to get container status \"1d2490d1278ee6eb8ab4e430d731848635d5fb526d2d655ea46d6f87c65a6e66\": rpc error: code = NotFound desc = could not find container \"1d2490d1278ee6eb8ab4e430d731848635d5fb526d2d655ea46d6f87c65a6e66\": container with ID starting with 1d2490d1278ee6eb8ab4e430d731848635d5fb526d2d655ea46d6f87c65a6e66 not found: ID does not exist"
Nov 02 23:23:28.896491 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:28.896404    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"]
Nov 02 23:23:28.898498 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:28.898470    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:28.943963 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:28.943931    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-tt2kk"]
Nov 02 23:23:29.058506 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:29.058479    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:30.549903 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:30.549870    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="602de6ee-5748-44d1-b389-7124d80e521b" path="/var/lib/kubelet/pods/602de6ee-5748-44d1-b389-7124d80e521b/volumes"
Nov 02 23:23:31.826790 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:31.826750    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:32.169850 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:32.169777    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:33.663472 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:33.663442    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j"
Nov 02 23:23:36.900300 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:36.900263    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:41.753007 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:23:41.752938    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:23:43.193047 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.193013    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"]
Nov 02 23:23:43.193652 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.193637    4294 cpu_manager.go:414] "containerMap: removing container" podUID="602de6ee-5748-44d1-b389-7124d80e521b" containerName="istio-proxy"
Nov 02 23:23:43.193706 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.193653    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="602de6ee-5748-44d1-b389-7124d80e521b" containerName="istio-proxy"
Nov 02 23:23:43.193779 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.193770    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="602de6ee-5748-44d1-b389-7124d80e521b" containerName="istio-proxy"
Nov 02 23:23:43.198321 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.198304    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"
Nov 02 23:23:43.298876 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.298836    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b57963-serving-cert\""
Nov 02 23:23:43.302016 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.301994    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"]
Nov 02 23:23:43.358518 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.358487    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhxzj\" (UniqueName: \"kubernetes.io/projected/300f6dfb-aeb3-4967-859b-664122971d83-kube-api-access-vhxzj\") pod \"maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9\" (UID: \"300f6dfb-aeb3-4967-859b-664122971d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"
Nov 02 23:23:43.358705 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.358543    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/300f6dfb-aeb3-4967-859b-664122971d83-metrics-certs\") pod \"maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9\" (UID: \"300f6dfb-aeb3-4967-859b-664122971d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"
Nov 02 23:23:43.358705 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.358680    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/300f6dfb-aeb3-4967-859b-664122971d83-maas-api-tls\") pod \"maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9\" (UID: \"300f6dfb-aeb3-4967-859b-664122971d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"
Nov 02 23:23:43.358840 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.358736    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/300f6dfb-aeb3-4967-859b-664122971d83-openshift-service-ca\") pod \"maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9\" (UID: \"300f6dfb-aeb3-4967-859b-664122971d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"
Nov 02 23:23:43.460604 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.460523    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhxzj\" (UniqueName: \"kubernetes.io/projected/300f6dfb-aeb3-4967-859b-664122971d83-kube-api-access-vhxzj\") pod \"maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9\" (UID: \"300f6dfb-aeb3-4967-859b-664122971d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"
Nov 02 23:23:43.460604 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.460583    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/300f6dfb-aeb3-4967-859b-664122971d83-metrics-certs\") pod \"maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9\" (UID: \"300f6dfb-aeb3-4967-859b-664122971d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"
Nov 02 23:23:43.460812 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.460623    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/300f6dfb-aeb3-4967-859b-664122971d83-maas-api-tls\") pod \"maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9\" (UID: \"300f6dfb-aeb3-4967-859b-664122971d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"
Nov 02 23:23:43.460812 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.460675    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/300f6dfb-aeb3-4967-859b-664122971d83-openshift-service-ca\") pod \"maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9\" (UID: \"300f6dfb-aeb3-4967-859b-664122971d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"
Nov 02 23:23:43.460812 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:23:43.460788    4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-serving-cert: secret "maas-api-e2e-api-a-b57963-serving-cert" not found
Nov 02 23:23:43.460925 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:23:43.460863    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/300f6dfb-aeb3-4967-859b-664122971d83-maas-api-tls podName:300f6dfb-aeb3-4967-859b-664122971d83 nodeName:}" failed. No retries permitted until 2026-11-02 23:23:43.96084346 +0000 UTC m=+1863.529198230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/300f6dfb-aeb3-4967-859b-664122971d83-maas-api-tls") pod "maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9" (UID: "300f6dfb-aeb3-4967-859b-664122971d83") : secret "maas-api-e2e-api-a-b57963-serving-cert" not found
Nov 02 23:23:43.461403 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.461376    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/300f6dfb-aeb3-4967-859b-664122971d83-openshift-service-ca\") pod \"maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9\" (UID: \"300f6dfb-aeb3-4967-859b-664122971d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"
Nov 02 23:23:43.463157 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.463131    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/300f6dfb-aeb3-4967-859b-664122971d83-metrics-certs\") pod \"maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9\" (UID: \"300f6dfb-aeb3-4967-859b-664122971d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"
Nov 02 23:23:43.528706 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.528674    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhxzj\" (UniqueName: \"kubernetes.io/projected/300f6dfb-aeb3-4967-859b-664122971d83-kube-api-access-vhxzj\") pod \"maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9\" (UID: \"300f6dfb-aeb3-4967-859b-664122971d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"
Nov 02 23:23:43.542111 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.542081    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:43.784452 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.784415    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:43.954874 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.954841    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:43.966337 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.966316    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/300f6dfb-aeb3-4967-859b-664122971d83-maas-api-tls\") pod \"maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9\" (UID: \"300f6dfb-aeb3-4967-859b-664122971d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"
Nov 02 23:23:43.968637 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:43.968611    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/300f6dfb-aeb3-4967-859b-664122971d83-maas-api-tls\") pod \"maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9\" (UID: \"300f6dfb-aeb3-4967-859b-664122971d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"
Nov 02 23:23:44.111402 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:44.111320    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"
Nov 02 23:23:44.482163 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:44.482127    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"]
Nov 02 23:23:44.482573 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:23:44.482221    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod300f6dfb_aeb3_4967_859b_664122971d83.slice/crio-3dd542a483fa09fe0b66053e5737cc99dff252bdc83b8f591b6b114b4be99466 WatchSource:0}: Error finding container 3dd542a483fa09fe0b66053e5737cc99dff252bdc83b8f591b6b114b4be99466: Status 404 returned error can't find the container with id 3dd542a483fa09fe0b66053e5737cc99dff252bdc83b8f591b6b114b4be99466
Nov 02 23:23:44.801674 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:44.801640    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9" event={"ID":"300f6dfb-aeb3-4967-859b-664122971d83","Type":"ContainerStarted","Data":"3dd542a483fa09fe0b66053e5737cc99dff252bdc83b8f591b6b114b4be99466"}
Nov 02 23:23:45.809217 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:45.809174    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9" event={"ID":"300f6dfb-aeb3-4967-859b-664122971d83","Type":"ContainerStarted","Data":"df8c1e06d3bdd0198f5577e5d41e7c80ac95751ca0dfc89730ea6692335451f1"}
Nov 02 23:23:45.809762 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:45.809250    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"
Nov 02 23:23:45.919775 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:45.919718    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9" podStartSLOduration=2.004361629 podStartE2EDuration="2.91970427s" totalImagesPullingTime="915.342641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:23:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:23:45.918565024 +0000 UTC m=+1865.486919812" watchObservedRunningTime="2026-11-02 23:23:45.91970427 +0000 UTC m=+1865.488059056"
Nov 02 23:23:48.474585 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:48.474546    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:48.814176 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:48.814146    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:48.978929 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:48.978899    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:51.821115 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:51.821090    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"
Nov 02 23:23:53.536522 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:53.536490    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:55.053000 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:55.052955    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq"]
Nov 02 23:23:55.153830 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:55.153791    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk"]
Nov 02 23:23:55.213154 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:55.213123    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:55.470276 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:55.470187    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:55.674414 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:55.674377    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:58.741776 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:58.741740    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:58.998227 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:58.998120    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:23:59.224162 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:23:59.224116    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:00.054207 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.054160    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq" podUID="5c71bd6a-003a-4245-a685-6b19be14c826" containerName="payload-processing" containerID="cri-o://1724442d717d7893cae28e15814e0316a11e3337ad78d804fa7ded0a738d8cf0" gracePeriod=25
Nov 02 23:24:00.154387 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.154341    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk" podUID="c2b82c9b-91dd-44e6-a474-4a80b62e916f" containerName="payload-pre-processing" containerID="cri-o://68af350dce90df9ab175004bf438136708635abda9d3b33a0b35b344bdf63a2a" gracePeriod=25
Nov 02 23:24:00.376865 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.376489    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq"
Nov 02 23:24:00.508972 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.508938    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c71bd6a-003a-4245-a685-6b19be14c826-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c71bd6a-003a-4245-a685-6b19be14c826-plugins-config-volume\") pod \"5c71bd6a-003a-4245-a685-6b19be14c826\" (UID: \"5c71bd6a-003a-4245-a685-6b19be14c826\") "
Nov 02 23:24:00.509214 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.509179    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c71bd6a-003a-4245-a685-6b19be14c826-kube-api-access-dbjs8\" (UniqueName: \"kubernetes.io/projected/5c71bd6a-003a-4245-a685-6b19be14c826-kube-api-access-dbjs8\") pod \"5c71bd6a-003a-4245-a685-6b19be14c826\" (UID: \"5c71bd6a-003a-4245-a685-6b19be14c826\") "
Nov 02 23:24:00.512122 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.510100    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c71bd6a-003a-4245-a685-6b19be14c826-plugins-config-volume" pod "5c71bd6a-003a-4245-a685-6b19be14c826" (UID: "5c71bd6a-003a-4245-a685-6b19be14c826"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:24:00.514385 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.514362    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c71bd6a-003a-4245-a685-6b19be14c826-kube-api-access-dbjs8" pod "5c71bd6a-003a-4245-a685-6b19be14c826" (UID: "5c71bd6a-003a-4245-a685-6b19be14c826"). InnerVolumeSpecName "kube-api-access-dbjs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:24:00.605249 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.605176    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk"
Nov 02 23:24:00.610934 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.610900    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbjs8\" (UniqueName: \"kubernetes.io/projected/5c71bd6a-003a-4245-a685-6b19be14c826-kube-api-access-dbjs8\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:24:00.610934 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.610935    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c71bd6a-003a-4245-a685-6b19be14c826-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:24:00.712761 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.712206    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b82c9b-91dd-44e6-a474-4a80b62e916f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b82c9b-91dd-44e6-a474-4a80b62e916f-plugins-config-volume\") pod \"c2b82c9b-91dd-44e6-a474-4a80b62e916f\" (UID: \"c2b82c9b-91dd-44e6-a474-4a80b62e916f\") "
Nov 02 23:24:00.712761 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.712336    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b82c9b-91dd-44e6-a474-4a80b62e916f-kube-api-access-9hlmc\" (UniqueName: \"kubernetes.io/projected/c2b82c9b-91dd-44e6-a474-4a80b62e916f-kube-api-access-9hlmc\") pod \"c2b82c9b-91dd-44e6-a474-4a80b62e916f\" (UID: \"c2b82c9b-91dd-44e6-a474-4a80b62e916f\") "
Nov 02 23:24:00.723661 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.713669    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b82c9b-91dd-44e6-a474-4a80b62e916f-plugins-config-volume" pod "c2b82c9b-91dd-44e6-a474-4a80b62e916f" (UID: "c2b82c9b-91dd-44e6-a474-4a80b62e916f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:24:00.737866 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.735198    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b82c9b-91dd-44e6-a474-4a80b62e916f-kube-api-access-9hlmc" pod "c2b82c9b-91dd-44e6-a474-4a80b62e916f" (UID: "c2b82c9b-91dd-44e6-a474-4a80b62e916f"). InnerVolumeSpecName "kube-api-access-9hlmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:24:00.814290 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.814237    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b82c9b-91dd-44e6-a474-4a80b62e916f-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:24:00.814290 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.814289    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hlmc\" (UniqueName: \"kubernetes.io/projected/c2b82c9b-91dd-44e6-a474-4a80b62e916f-kube-api-access-9hlmc\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:24:00.906307 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.906204    4294 generic.go:356] "Generic (PLEG): container finished" podID="c2b82c9b-91dd-44e6-a474-4a80b62e916f" containerID="68af350dce90df9ab175004bf438136708635abda9d3b33a0b35b344bdf63a2a" exitCode=0
Nov 02 23:24:00.906482 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.906398    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk" event={"ID":"c2b82c9b-91dd-44e6-a474-4a80b62e916f","Type":"ContainerDied","Data":"68af350dce90df9ab175004bf438136708635abda9d3b33a0b35b344bdf63a2a"}
Nov 02 23:24:00.906482 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.906451    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk"
Nov 02 23:24:00.906600 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.906481    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk" event={"ID":"c2b82c9b-91dd-44e6-a474-4a80b62e916f","Type":"ContainerDied","Data":"2630aade0398744a2ddaedb83ae8a235061a640ad631b4ee4c55f32768b8a61c"}
Nov 02 23:24:00.906600 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.906510    4294 scope.go:122] "RemoveContainer" containerID="68af350dce90df9ab175004bf438136708635abda9d3b33a0b35b344bdf63a2a"
Nov 02 23:24:00.908092 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.908068    4294 generic.go:356] "Generic (PLEG): container finished" podID="5c71bd6a-003a-4245-a685-6b19be14c826" containerID="1724442d717d7893cae28e15814e0316a11e3337ad78d804fa7ded0a738d8cf0" exitCode=0
Nov 02 23:24:00.908236 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.908164    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq" event={"ID":"5c71bd6a-003a-4245-a685-6b19be14c826","Type":"ContainerDied","Data":"1724442d717d7893cae28e15814e0316a11e3337ad78d804fa7ded0a738d8cf0"}
Nov 02 23:24:00.908236 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.908199    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq" event={"ID":"5c71bd6a-003a-4245-a685-6b19be14c826","Type":"ContainerDied","Data":"79657fdb6e08a827f467d9c76570409a9f930b7be073ada9ab388778112c473b"}
Nov 02 23:24:00.908486 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.908462    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq"
Nov 02 23:24:00.921549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.919408    4294 scope.go:122] "RemoveContainer" containerID="68af350dce90df9ab175004bf438136708635abda9d3b33a0b35b344bdf63a2a"
Nov 02 23:24:00.922506 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:24:00.922479    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68af350dce90df9ab175004bf438136708635abda9d3b33a0b35b344bdf63a2a\": container with ID starting with 68af350dce90df9ab175004bf438136708635abda9d3b33a0b35b344bdf63a2a not found: ID does not exist" containerID="68af350dce90df9ab175004bf438136708635abda9d3b33a0b35b344bdf63a2a"
Nov 02 23:24:00.922595 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.922515    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68af350dce90df9ab175004bf438136708635abda9d3b33a0b35b344bdf63a2a"} err="failed to get container status \"68af350dce90df9ab175004bf438136708635abda9d3b33a0b35b344bdf63a2a\": rpc error: code = NotFound desc = could not find container \"68af350dce90df9ab175004bf438136708635abda9d3b33a0b35b344bdf63a2a\": container with ID starting with 68af350dce90df9ab175004bf438136708635abda9d3b33a0b35b344bdf63a2a not found: ID does not exist"
Nov 02 23:24:00.922595 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.922538    4294 scope.go:122] "RemoveContainer" containerID="1724442d717d7893cae28e15814e0316a11e3337ad78d804fa7ded0a738d8cf0"
Nov 02 23:24:00.933101 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.933075    4294 scope.go:122] "RemoveContainer" containerID="1724442d717d7893cae28e15814e0316a11e3337ad78d804fa7ded0a738d8cf0"
Nov 02 23:24:00.933453 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:24:00.933428    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1724442d717d7893cae28e15814e0316a11e3337ad78d804fa7ded0a738d8cf0\": container with ID starting with 1724442d717d7893cae28e15814e0316a11e3337ad78d804fa7ded0a738d8cf0 not found: ID does not exist" containerID="1724442d717d7893cae28e15814e0316a11e3337ad78d804fa7ded0a738d8cf0"
Nov 02 23:24:00.933497 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:00.933466    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1724442d717d7893cae28e15814e0316a11e3337ad78d804fa7ded0a738d8cf0"} err="failed to get container status \"1724442d717d7893cae28e15814e0316a11e3337ad78d804fa7ded0a738d8cf0\": rpc error: code = NotFound desc = could not find container \"1724442d717d7893cae28e15814e0316a11e3337ad78d804fa7ded0a738d8cf0\": container with ID starting with 1724442d717d7893cae28e15814e0316a11e3337ad78d804fa7ded0a738d8cf0 not found: ID does not exist"
Nov 02 23:24:01.066160 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:01.066124    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk"]
Nov 02 23:24:01.073473 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:01.073443    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6a894c-67dd6bf855-5pbsk"]
Nov 02 23:24:01.180758 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:01.180653    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq"]
Nov 02 23:24:01.183731 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:01.183701    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6a894c-74cbd67495-rfbxq"]
Nov 02 23:24:02.547756 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:02.547721    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c71bd6a-003a-4245-a685-6b19be14c826" path="/var/lib/kubelet/pods/5c71bd6a-003a-4245-a685-6b19be14c826/volumes"
Nov 02 23:24:02.548176 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:02.548161    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b82c9b-91dd-44e6-a474-4a80b62e916f" path="/var/lib/kubelet/pods/c2b82c9b-91dd-44e6-a474-4a80b62e916f/volumes"
Nov 02 23:24:03.874242 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:03.874212    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:08.664761 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:08.664724    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:08.828183 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:08.828146    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:11.749252 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:24:11.749218    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:24:13.674045 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:13.674005    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:18.579567 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:18.579530    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:18.713435 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:18.713400    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"]
Nov 02 23:24:18.713645 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:18.713621    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg" podUID="8a3538a4-c25e-4d3d-97e5-267a770bbb43" containerName="istio-proxy" containerID="cri-o://402ec30c6b2c75c44c115e4b7a438712813a599b5ae1a25989bc373a7e80b004" gracePeriod=30
Nov 02 23:24:22.307660 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:22.307622    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 23:24:22.308064 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:22.307687    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg" podUID="8a3538a4-c25e-4d3d-97e5-267a770bbb43" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 23:24:23.501739 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.501693    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"]
Nov 02 23:24:23.502813 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.502595    4294 cpu_manager.go:414] "containerMap: removing container" podUID="c2b82c9b-91dd-44e6-a474-4a80b62e916f" containerName="payload-pre-processing"
Nov 02 23:24:23.502813 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.502618    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b82c9b-91dd-44e6-a474-4a80b62e916f" containerName="payload-pre-processing"
Nov 02 23:24:23.502813 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.502677    4294 cpu_manager.go:414] "containerMap: removing container" podUID="5c71bd6a-003a-4245-a685-6b19be14c826" containerName="payload-processing"
Nov 02 23:24:23.502813 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.502686    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c71bd6a-003a-4245-a685-6b19be14c826" containerName="payload-processing"
Nov 02 23:24:23.503082 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.502837    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c71bd6a-003a-4245-a685-6b19be14c826" containerName="payload-processing"
Nov 02 23:24:23.503082 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.502859    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b82c9b-91dd-44e6-a474-4a80b62e916f" containerName="payload-pre-processing"
Nov 02 23:24:23.506478 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.506457    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"
Nov 02 23:24:23.561162 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.561121    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"]
Nov 02 23:24:23.585739 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.585701    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-981724-serving-cert\""
Nov 02 23:24:23.586152 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.586128    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4\" (UID: \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"
Nov 02 23:24:23.586343 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.586326    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4\" (UID: \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"
Nov 02 23:24:23.586416 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.586377    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-metrics-certs\") pod \"maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4\" (UID: \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"
Nov 02 23:24:23.586475 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.586437    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slrdb\" (UniqueName: \"kubernetes.io/projected/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-kube-api-access-slrdb\") pod \"maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4\" (UID: \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"
Nov 02 23:24:23.646784 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.646739    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:23.688132 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.688096    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4\" (UID: \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"
Nov 02 23:24:23.688321 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.688155    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-metrics-certs\") pod \"maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4\" (UID: \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"
Nov 02 23:24:23.688321 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.688183    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slrdb\" (UniqueName: \"kubernetes.io/projected/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-kube-api-access-slrdb\") pod \"maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4\" (UID: \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"
Nov 02 23:24:23.688321 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.688206    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4\" (UID: \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"
Nov 02 23:24:23.688321 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:24:23.688245    4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-serving-cert: secret "maas-api-e2e-ipp-a-981724-serving-cert" not found
Nov 02 23:24:23.688321 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:24:23.688319    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-maas-api-tls podName:f3299cc5-a6a4-4cf1-9603-e14541a1d92e nodeName:}" failed. No retries permitted until 2026-11-02 23:24:24.188297854 +0000 UTC m=+1903.756652620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-maas-api-tls") pod "maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4" (UID: "f3299cc5-a6a4-4cf1-9603-e14541a1d92e") : secret "maas-api-e2e-ipp-a-981724-serving-cert" not found
Nov 02 23:24:23.688884 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.688867    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4\" (UID: \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"
Nov 02 23:24:23.690544 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.690519    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-metrics-certs\") pod \"maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4\" (UID: \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"
Nov 02 23:24:23.745250 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.745226    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slrdb\" (UniqueName: \"kubernetes.io/projected/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-kube-api-access-slrdb\") pod \"maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4\" (UID: \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"
Nov 02 23:24:23.822854 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.822777    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:23.984332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.984305    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:24:23.996112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:23.996086    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:24.058081 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.058046    4294 generic.go:356] "Generic (PLEG): container finished" podID="8a3538a4-c25e-4d3d-97e5-267a770bbb43" containerID="402ec30c6b2c75c44c115e4b7a438712813a599b5ae1a25989bc373a7e80b004" exitCode=0
Nov 02 23:24:24.058277 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.058094    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"
Nov 02 23:24:24.058277 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.058100    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg" event={"ID":"8a3538a4-c25e-4d3d-97e5-267a770bbb43","Type":"ContainerDied","Data":"402ec30c6b2c75c44c115e4b7a438712813a599b5ae1a25989bc373a7e80b004"}
Nov 02 23:24:24.058277 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.058164    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg" event={"ID":"8a3538a4-c25e-4d3d-97e5-267a770bbb43","Type":"ContainerDied","Data":"8705820a7bd01ee1d31ba667d615c40cc2e7e2dda59972785d9deebd0ae78d30"}
Nov 02 23:24:24.058277 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.058187    4294 scope.go:122] "RemoveContainer" containerID="402ec30c6b2c75c44c115e4b7a438712813a599b5ae1a25989bc373a7e80b004"
Nov 02 23:24:24.068197 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.068177    4294 scope.go:122] "RemoveContainer" containerID="402ec30c6b2c75c44c115e4b7a438712813a599b5ae1a25989bc373a7e80b004"
Nov 02 23:24:24.068461 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:24:24.068443    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"402ec30c6b2c75c44c115e4b7a438712813a599b5ae1a25989bc373a7e80b004\": container with ID starting with 402ec30c6b2c75c44c115e4b7a438712813a599b5ae1a25989bc373a7e80b004 not found: ID does not exist" containerID="402ec30c6b2c75c44c115e4b7a438712813a599b5ae1a25989bc373a7e80b004"
Nov 02 23:24:24.068508 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.068470    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"402ec30c6b2c75c44c115e4b7a438712813a599b5ae1a25989bc373a7e80b004"} err="failed to get container status \"402ec30c6b2c75c44c115e4b7a438712813a599b5ae1a25989bc373a7e80b004\": rpc error: code = NotFound desc = could not find container \"402ec30c6b2c75c44c115e4b7a438712813a599b5ae1a25989bc373a7e80b004\": container with ID starting with 402ec30c6b2c75c44c115e4b7a438712813a599b5ae1a25989bc373a7e80b004 not found: ID does not exist"
Nov 02 23:24:24.097930 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.097845    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istiod-ca-cert\") pod \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") "
Nov 02 23:24:24.097930 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.097917    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-credential-socket\") pod \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") "
Nov 02 23:24:24.098205 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.097954    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-workload-socket\") pod \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") "
Nov 02 23:24:24.098205 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.098066    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-token\" (UniqueName: \"kubernetes.io/projected/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-token\") pod \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") "
Nov 02 23:24:24.098205 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.098166    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-podinfo\") pod \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") "
Nov 02 23:24:24.098205 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.098198    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3538a4-c25e-4d3d-97e5-267a770bbb43-kube-api-access-zkprk\" (UniqueName: \"kubernetes.io/projected/8a3538a4-c25e-4d3d-97e5-267a770bbb43-kube-api-access-zkprk\") pod \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") "
Nov 02 23:24:24.098446 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.098233    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-data\") pod \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") "
Nov 02 23:24:24.098446 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.098293    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-workload-certs\") pod \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") "
Nov 02 23:24:24.098446 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.098365    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-credential-socket" pod "8a3538a4-c25e-4d3d-97e5-267a770bbb43" (UID: "8a3538a4-c25e-4d3d-97e5-267a770bbb43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:24:24.098446 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.098378    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istiod-ca-cert" pod "8a3538a4-c25e-4d3d-97e5-267a770bbb43" (UID: "8a3538a4-c25e-4d3d-97e5-267a770bbb43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:24:24.098666 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.098579    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-workload-socket" pod "8a3538a4-c25e-4d3d-97e5-267a770bbb43" (UID: "8a3538a4-c25e-4d3d-97e5-267a770bbb43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:24:24.098666 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.098632    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-workload-certs" pod "8a3538a4-c25e-4d3d-97e5-267a770bbb43" (UID: "8a3538a4-c25e-4d3d-97e5-267a770bbb43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:24:24.098901 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.098879    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-envoy\") pod \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\" (UID: \"8a3538a4-c25e-4d3d-97e5-267a770bbb43\") "
Nov 02 23:24:24.099019 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.098884    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-data" pod "8a3538a4-c25e-4d3d-97e5-267a770bbb43" (UID: "8a3538a4-c25e-4d3d-97e5-267a770bbb43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:24:24.099495 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.099466    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:24:24.099495 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.099490    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:24:24.099647 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.099504    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:24:24.099647 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.099520    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:24:24.099647 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.099535    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:24:24.101025 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.100962    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-token" pod "8a3538a4-c25e-4d3d-97e5-267a770bbb43" (UID: "8a3538a4-c25e-4d3d-97e5-267a770bbb43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:24:24.101025 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.101000    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-podinfo" pod "8a3538a4-c25e-4d3d-97e5-267a770bbb43" (UID: "8a3538a4-c25e-4d3d-97e5-267a770bbb43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:24:24.101025 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.100962    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-envoy" pod "8a3538a4-c25e-4d3d-97e5-267a770bbb43" (UID: "8a3538a4-c25e-4d3d-97e5-267a770bbb43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:24:24.101281 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.101261    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3538a4-c25e-4d3d-97e5-267a770bbb43-kube-api-access-zkprk" pod "8a3538a4-c25e-4d3d-97e5-267a770bbb43" (UID: "8a3538a4-c25e-4d3d-97e5-267a770bbb43"). InnerVolumeSpecName "kube-api-access-zkprk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:24:24.160722 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.160684    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:24.201262 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.201223    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4\" (UID: \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"
Nov 02 23:24:24.201440 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.201329    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:24:24.201440 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.201340    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:24:24.201440 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.201349    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a3538a4-c25e-4d3d-97e5-267a770bbb43-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:24:24.201440 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.201359    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkprk\" (UniqueName: \"kubernetes.io/projected/8a3538a4-c25e-4d3d-97e5-267a770bbb43-kube-api-access-zkprk\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:24:24.203654 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.203627    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4\" (UID: \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"
Nov 02 23:24:24.420868 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.420837    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"
Nov 02 23:24:24.478211 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.477551    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"]
Nov 02 23:24:24.523478 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.523448    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-p25cg"]
Nov 02 23:24:24.551652 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.551613    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a3538a4-c25e-4d3d-97e5-267a770bbb43" path="/var/lib/kubelet/pods/8a3538a4-c25e-4d3d-97e5-267a770bbb43/volumes"
Nov 02 23:24:24.795709 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:24:24.795676    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3299cc5_a6a4_4cf1_9603_e14541a1d92e.slice/crio-e3fabda7211a50cad590d809f4cab8777228a20a6d476b4c7ddfe1c5f2ec740b WatchSource:0}: Error finding container e3fabda7211a50cad590d809f4cab8777228a20a6d476b4c7ddfe1c5f2ec740b: Status 404 returned error can't find the container with id e3fabda7211a50cad590d809f4cab8777228a20a6d476b4c7ddfe1c5f2ec740b
Nov 02 23:24:24.795872 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:24.795852    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"]
Nov 02 23:24:25.065116 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:25.065083    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4" event={"ID":"f3299cc5-a6a4-4cf1-9603-e14541a1d92e","Type":"ContainerStarted","Data":"e3fabda7211a50cad590d809f4cab8777228a20a6d476b4c7ddfe1c5f2ec740b"}
Nov 02 23:24:25.841182 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:25.841129    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:26.013655 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:26.013593    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:26.074568 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:26.074534    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4" event={"ID":"f3299cc5-a6a4-4cf1-9603-e14541a1d92e","Type":"ContainerStarted","Data":"de8fc5d67baaef3aed68d5c6ec77c8b691b3661ccfa0d0189db7a660e08450fa"}
Nov 02 23:24:26.074747 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:26.074632    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"
Nov 02 23:24:26.187336 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:26.187275    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4" podStartSLOduration=2.363175017 podStartE2EDuration="3.187254879s" totalImagesPullingTime="824.079862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:24:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:24:26.184080327 +0000 UTC m=+1905.752435126" watchObservedRunningTime="2026-11-02 23:24:26.187254879 +0000 UTC m=+1905.755609667"
Nov 02 23:24:28.912919 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:28.912882    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:31.502789 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:31.502751    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:32.094598 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:32.094567    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"
Nov 02 23:24:32.230898 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:32.230865    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:32.395512 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:32.395432    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:34.241419 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:34.241384    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:36.152214 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:36.152181    4294 generic.go:356] "Generic (PLEG): container finished" podID="c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" containerID="39e32cfa3308105a603e7508711f246bc3206b5a8d787e290bf5efe8d27f176e" exitCode=0
Nov 02 23:24:36.152702 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:36.152259    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb" event={"ID":"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2","Type":"ContainerDied","Data":"39e32cfa3308105a603e7508711f246bc3206b5a8d787e290bf5efe8d27f176e"}
Nov 02 23:24:36.152702 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:36.152681    4294 scope.go:122] "RemoveContainer" containerID="39e32cfa3308105a603e7508711f246bc3206b5a8d787e290bf5efe8d27f176e"
Nov 02 23:24:36.154171 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:36.154142    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:24:36.154270 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:36.154210    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:24:36.154270 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:36.154241    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:24:37.159686 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:37.159654    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb" event={"ID":"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2","Type":"ContainerStarted","Data":"5092e4f738ec88a88dd8411d47dcfc98a7787e6f0bd865504cc47f08112fd1a6"}
Nov 02 23:24:37.697529 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:37.697492    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:24:37.698606 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:37.698577    4294 patch_prober.go:29] interesting pod/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 23:24:37.698743 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:37.698624    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb" podUID="c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 23:24:38.446038 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:38.446007    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:24:38.698092 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:38.697974    4294 patch_prober.go:29] interesting pod/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 23:24:38.698092 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:38.698067    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb" podUID="c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 23:24:39.697643 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:39.697600    4294 patch_prober.go:29] interesting pod/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 23:24:39.698121 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:39.697669    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb" podUID="c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 23:24:40.697626 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:40.697589    4294 patch_prober.go:29] interesting pod/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 23:24:40.697844 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:40.697669    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb" podUID="c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 23:24:41.533385 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:41.533348    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:41.698193 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:41.698058    4294 patch_prober.go:29] interesting pod/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 23:24:41.698193 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:41.698121    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb" podUID="c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 23:24:41.789785 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:24:41.789520    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:24:42.698245 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:42.698206    4294 patch_prober.go:29] interesting pod/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 02 23:24:42.698654 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:42.698273    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb" podUID="c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 02 23:24:42.892755 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:42.890461    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:43.702700 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:43.702668    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:24:43.703770 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:43.703752    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:24:45.905345 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:45.905302    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:46.269248 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:46.269204    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv"]
Nov 02 23:24:46.269447 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:46.269423    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv" podUID="14520749-ad0f-4518-a370-e5559f3fd565" containerName="payload-processing" containerID="cri-o://8f023eee4dad8c5d4ee3433932f152fdd6dd42332511508091baa04aea34ea7f" gracePeriod=30
Nov 02 23:24:46.502668 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:46.502633    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:46.547889 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:46.547865    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv"
Nov 02 23:24:46.638662 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:46.638627    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14520749-ad0f-4518-a370-e5559f3fd565-kube-api-access-wbgtc\" (UniqueName: \"kubernetes.io/projected/14520749-ad0f-4518-a370-e5559f3fd565-kube-api-access-wbgtc\") pod \"14520749-ad0f-4518-a370-e5559f3fd565\" (UID: \"14520749-ad0f-4518-a370-e5559f3fd565\") "
Nov 02 23:24:46.638852 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:46.638678    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14520749-ad0f-4518-a370-e5559f3fd565-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14520749-ad0f-4518-a370-e5559f3fd565-plugins-config-volume\") pod \"14520749-ad0f-4518-a370-e5559f3fd565\" (UID: \"14520749-ad0f-4518-a370-e5559f3fd565\") "
Nov 02 23:24:46.639166 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:46.639145    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14520749-ad0f-4518-a370-e5559f3fd565-plugins-config-volume" pod "14520749-ad0f-4518-a370-e5559f3fd565" (UID: "14520749-ad0f-4518-a370-e5559f3fd565"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:24:46.639438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:46.639415    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14520749-ad0f-4518-a370-e5559f3fd565-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:24:46.640764 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:46.640743    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14520749-ad0f-4518-a370-e5559f3fd565-kube-api-access-wbgtc" pod "14520749-ad0f-4518-a370-e5559f3fd565" (UID: "14520749-ad0f-4518-a370-e5559f3fd565"). InnerVolumeSpecName "kube-api-access-wbgtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:24:46.741162 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:46.741129    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbgtc\" (UniqueName: \"kubernetes.io/projected/14520749-ad0f-4518-a370-e5559f3fd565-kube-api-access-wbgtc\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:24:46.752779 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:46.752748    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:47.234026 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:47.233991    4294 generic.go:356] "Generic (PLEG): container finished" podID="14520749-ad0f-4518-a370-e5559f3fd565" containerID="8f023eee4dad8c5d4ee3433932f152fdd6dd42332511508091baa04aea34ea7f" exitCode=0
Nov 02 23:24:47.234438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:47.234049    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv"
Nov 02 23:24:47.234438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:47.234080    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv" event={"ID":"14520749-ad0f-4518-a370-e5559f3fd565","Type":"ContainerDied","Data":"8f023eee4dad8c5d4ee3433932f152fdd6dd42332511508091baa04aea34ea7f"}
Nov 02 23:24:47.234438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:47.234152    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv" event={"ID":"14520749-ad0f-4518-a370-e5559f3fd565","Type":"ContainerDied","Data":"26e1fc2acfd123119e1dcb747158e77f447a129da07e7c74d6872f91a775e042"}
Nov 02 23:24:47.234438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:47.234173    4294 scope.go:122] "RemoveContainer" containerID="8f023eee4dad8c5d4ee3433932f152fdd6dd42332511508091baa04aea34ea7f"
Nov 02 23:24:47.243938 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:47.243920    4294 scope.go:122] "RemoveContainer" containerID="8f023eee4dad8c5d4ee3433932f152fdd6dd42332511508091baa04aea34ea7f"
Nov 02 23:24:47.244198 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:24:47.244179    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f023eee4dad8c5d4ee3433932f152fdd6dd42332511508091baa04aea34ea7f\": container with ID starting with 8f023eee4dad8c5d4ee3433932f152fdd6dd42332511508091baa04aea34ea7f not found: ID does not exist" containerID="8f023eee4dad8c5d4ee3433932f152fdd6dd42332511508091baa04aea34ea7f"
Nov 02 23:24:47.244243 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:47.244208    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f023eee4dad8c5d4ee3433932f152fdd6dd42332511508091baa04aea34ea7f"} err="failed to get container status \"8f023eee4dad8c5d4ee3433932f152fdd6dd42332511508091baa04aea34ea7f\": rpc error: code = NotFound desc = could not find container \"8f023eee4dad8c5d4ee3433932f152fdd6dd42332511508091baa04aea34ea7f\": container with ID starting with 8f023eee4dad8c5d4ee3433932f152fdd6dd42332511508091baa04aea34ea7f not found: ID does not exist"
Nov 02 23:24:47.391293 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:47.391259    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv"]
Nov 02 23:24:47.438656 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:47.438630    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-027689-7f8f76c546-bbqlv"]
Nov 02 23:24:47.936242 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:47.936211    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:48.548170 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:48.548142    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14520749-ad0f-4518-a370-e5559f3fd565" path="/var/lib/kubelet/pods/14520749-ad0f-4518-a370-e5559f3fd565/volumes"
Nov 02 23:24:52.990852 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:52.990798    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:54.395354 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:54.395281    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:54.605689 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:54.605643    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:55.933491 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:55.933456    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:56.579721 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:56.579689    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:57.020510 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:57.020476    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:57.348944 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:57.348859    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:57.594415 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:57.594384    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:57.750880 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:57.750851    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:58.425392 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:58.425354    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:59.124626 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:59.124594    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:24:59.505410 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:24:59.505376    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:00.511174 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:00.511140    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:03.106819 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:03.106781    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:03.587623 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:03.587589    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:03.767532 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:03.767496    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:03.940540 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:03.940455    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:05.876106 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:05.876072    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:07.338718 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:07.338679    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:07.517479 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:07.517448    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:11.227789 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:11.227746    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:11.749444 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:25:11.749410    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:25:12.502693 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:12.502652    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:12.832597 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:12.832518    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t"]
Nov 02 23:25:13.234144 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:13.234064    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:13.431975 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:13.431940    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:14.020096 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:14.020059    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:14.238503 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:14.238464    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:14.424328 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:14.424243    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:14.738523 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:14.738449    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:14.916857 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:14.916820    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:16.664950 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:16.664915    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:17.833325 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:17.833267    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t" podUID="6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e" containerName="payload-processing" containerID="cri-o://cc8ead1cec3c14cde7cc1ca02614f9c5222e4468edee064b9387cb0acde50b64" gracePeriod=25
Nov 02 23:25:18.094471 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:18.094413    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t"
Nov 02 23:25:18.144167 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:18.144132    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e-kube-api-access-fbmnn\" (UniqueName: \"kubernetes.io/projected/6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e-kube-api-access-fbmnn\") pod \"6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e\" (UID: \"6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e\") "
Nov 02 23:25:18.144331 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:18.144316    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e-plugins-config-volume\") pod \"6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e\" (UID: \"6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e\") "
Nov 02 23:25:18.144742 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:18.144713    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e-plugins-config-volume" pod "6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e" (UID: "6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:25:18.146266 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:18.146242    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e-kube-api-access-fbmnn" pod "6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e" (UID: "6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e"). InnerVolumeSpecName "kube-api-access-fbmnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:25:18.245735 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:18.245693    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbmnn\" (UniqueName: \"kubernetes.io/projected/6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e-kube-api-access-fbmnn\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:25:18.245735 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:18.245730    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:25:18.416785 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:18.416695    4294 generic.go:356] "Generic (PLEG): container finished" podID="6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e" containerID="cc8ead1cec3c14cde7cc1ca02614f9c5222e4468edee064b9387cb0acde50b64" exitCode=0
Nov 02 23:25:18.416785 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:18.416738    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t"
Nov 02 23:25:18.417069 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:18.416779    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t" event={"ID":"6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e","Type":"ContainerDied","Data":"cc8ead1cec3c14cde7cc1ca02614f9c5222e4468edee064b9387cb0acde50b64"}
Nov 02 23:25:18.417069 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:18.416840    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t" event={"ID":"6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e","Type":"ContainerDied","Data":"6bee9efa294879b3102aae57e7e3e8ff6971bf18e7e37e6c5b0463789903ec57"}
Nov 02 23:25:18.417069 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:18.416870    4294 scope.go:122] "RemoveContainer" containerID="cc8ead1cec3c14cde7cc1ca02614f9c5222e4468edee064b9387cb0acde50b64"
Nov 02 23:25:18.429431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:18.429400    4294 scope.go:122] "RemoveContainer" containerID="cc8ead1cec3c14cde7cc1ca02614f9c5222e4468edee064b9387cb0acde50b64"
Nov 02 23:25:18.429928 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:25:18.429897    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc8ead1cec3c14cde7cc1ca02614f9c5222e4468edee064b9387cb0acde50b64\": container with ID starting with cc8ead1cec3c14cde7cc1ca02614f9c5222e4468edee064b9387cb0acde50b64 not found: ID does not exist" containerID="cc8ead1cec3c14cde7cc1ca02614f9c5222e4468edee064b9387cb0acde50b64"
Nov 02 23:25:18.430024 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:18.429941    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc8ead1cec3c14cde7cc1ca02614f9c5222e4468edee064b9387cb0acde50b64"} err="failed to get container status \"cc8ead1cec3c14cde7cc1ca02614f9c5222e4468edee064b9387cb0acde50b64\": rpc error: code = NotFound desc = could not find container \"cc8ead1cec3c14cde7cc1ca02614f9c5222e4468edee064b9387cb0acde50b64\": container with ID starting with cc8ead1cec3c14cde7cc1ca02614f9c5222e4468edee064b9387cb0acde50b64 not found: ID does not exist"
Nov 02 23:25:18.440223 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:18.440201    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:18.579414 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:18.579367    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t"]
Nov 02 23:25:18.582545 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:18.582514    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9f5790-676bbdfbd7-4bc2t"]
Nov 02 23:25:18.612131 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:18.612098    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:18.786585 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:18.786557    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:19.948035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:19.947997    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:20.547528 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:20.547495    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e" path="/var/lib/kubelet/pods/6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e/volumes"
Nov 02 23:25:21.198169 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:21.198135    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:21.736646 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:21.736613    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:21.941361 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:21.941329    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:23.216671 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:23.216631    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:26.294091 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:26.294061    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:26.456572 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:26.456528    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:27.838765 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:27.838735    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:28.332471 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:28.332433    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:28.513768 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:28.513739    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:29.801795 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:29.801747    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:30.377374 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:30.377337    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:30.579945 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:30.579915    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:32.557224 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:32.557170    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:33.734561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:33.734528    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:35.117009 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:35.116966    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:35.146294 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:35.146263    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j"]
Nov 02 23:25:35.526747 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:35.526710    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:35.697276 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:35.697227    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:35.918837 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:35.918744    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:36.094255 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:36.094225    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:36.272822 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:36.272782    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:38.332625 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:38.332591    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:38.901750 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:38.901705    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:39.057531 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:39.057497    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:39.277415 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:39.277376    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:39.435365 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:39.435330    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:40.147553 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:40.147511    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j" podUID="46b473b0-75fd-4d11-8670-3ffd0abb90ac" containerName="payload-processing" containerID="cri-o://bd6dee7988c11ff5d07a381d985caec12a4972cb011bc49371a67fe40f3ba3b5" gracePeriod=25
Nov 02 23:25:40.310077 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:40.310046    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:40.417623 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:40.417600    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j"
Nov 02 23:25:40.492114 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:40.492084    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b473b0-75fd-4d11-8670-3ffd0abb90ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b473b0-75fd-4d11-8670-3ffd0abb90ac-plugins-config-volume\") pod \"46b473b0-75fd-4d11-8670-3ffd0abb90ac\" (UID: \"46b473b0-75fd-4d11-8670-3ffd0abb90ac\") "
Nov 02 23:25:40.492470 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:40.492223    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b473b0-75fd-4d11-8670-3ffd0abb90ac-kube-api-access-qltrl\" (UniqueName: \"kubernetes.io/projected/46b473b0-75fd-4d11-8670-3ffd0abb90ac-kube-api-access-qltrl\") pod \"46b473b0-75fd-4d11-8670-3ffd0abb90ac\" (UID: \"46b473b0-75fd-4d11-8670-3ffd0abb90ac\") "
Nov 02 23:25:40.492470 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:40.492430    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b473b0-75fd-4d11-8670-3ffd0abb90ac-plugins-config-volume" pod "46b473b0-75fd-4d11-8670-3ffd0abb90ac" (UID: "46b473b0-75fd-4d11-8670-3ffd0abb90ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:25:40.492655 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:40.492638    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b473b0-75fd-4d11-8670-3ffd0abb90ac-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:25:40.494281 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:40.494259    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b473b0-75fd-4d11-8670-3ffd0abb90ac-kube-api-access-qltrl" pod "46b473b0-75fd-4d11-8670-3ffd0abb90ac" (UID: "46b473b0-75fd-4d11-8670-3ffd0abb90ac"). InnerVolumeSpecName "kube-api-access-qltrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:25:40.566796 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:40.566765    4294 generic.go:356] "Generic (PLEG): container finished" podID="46b473b0-75fd-4d11-8670-3ffd0abb90ac" containerID="bd6dee7988c11ff5d07a381d985caec12a4972cb011bc49371a67fe40f3ba3b5" exitCode=0
Nov 02 23:25:40.566958 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:40.566803    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j"
Nov 02 23:25:40.566958 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:40.566853    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j" event={"ID":"46b473b0-75fd-4d11-8670-3ffd0abb90ac","Type":"ContainerDied","Data":"bd6dee7988c11ff5d07a381d985caec12a4972cb011bc49371a67fe40f3ba3b5"}
Nov 02 23:25:40.566958 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:40.566913    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j" event={"ID":"46b473b0-75fd-4d11-8670-3ffd0abb90ac","Type":"ContainerDied","Data":"323ed99d02d88cdc85c08547becbae5c365c03d7068c20b7f013ec0b7fcedb84"}
Nov 02 23:25:40.567155 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:40.566944    4294 scope.go:122] "RemoveContainer" containerID="bd6dee7988c11ff5d07a381d985caec12a4972cb011bc49371a67fe40f3ba3b5"
Nov 02 23:25:40.576737 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:40.576719    4294 scope.go:122] "RemoveContainer" containerID="bd6dee7988c11ff5d07a381d985caec12a4972cb011bc49371a67fe40f3ba3b5"
Nov 02 23:25:40.577032 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:25:40.577013    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6dee7988c11ff5d07a381d985caec12a4972cb011bc49371a67fe40f3ba3b5\": container with ID starting with bd6dee7988c11ff5d07a381d985caec12a4972cb011bc49371a67fe40f3ba3b5 not found: ID does not exist" containerID="bd6dee7988c11ff5d07a381d985caec12a4972cb011bc49371a67fe40f3ba3b5"
Nov 02 23:25:40.577106 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:40.577039    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6dee7988c11ff5d07a381d985caec12a4972cb011bc49371a67fe40f3ba3b5"} err="failed to get container status \"bd6dee7988c11ff5d07a381d985caec12a4972cb011bc49371a67fe40f3ba3b5\": rpc error: code = NotFound desc = could not find container \"bd6dee7988c11ff5d07a381d985caec12a4972cb011bc49371a67fe40f3ba3b5\": container with ID starting with bd6dee7988c11ff5d07a381d985caec12a4972cb011bc49371a67fe40f3ba3b5 not found: ID does not exist"
Nov 02 23:25:40.594322 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:40.594260    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qltrl\" (UniqueName: \"kubernetes.io/projected/46b473b0-75fd-4d11-8670-3ffd0abb90ac-kube-api-access-qltrl\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:25:40.724474 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:40.724373    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j"]
Nov 02 23:25:40.729284 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:40.729258    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-981724-86795b7dff-gq96j"]
Nov 02 23:25:41.752891 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:25:41.752848    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:25:42.548410 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:42.548376    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b473b0-75fd-4d11-8670-3ffd0abb90ac" path="/var/lib/kubelet/pods/46b473b0-75fd-4d11-8670-3ffd0abb90ac/volumes"
Nov 02 23:25:42.965810 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:42.965711    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:43.190268 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:43.190234    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:44.405916 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:44.405885    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:44.893102 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:44.893073    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:45.573532 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:45.573480    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:45.735127 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:45.735094    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:51.131335 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:51.131302    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:51.390893 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:51.390802    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:25:58.312719 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:25:58.312686    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:03.150615 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:03.150577    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:03.334098 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:03.334065    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:07.581905 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:07.581871    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:09.375682 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:09.375635    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:11.750961 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:26:11.750921    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:26:14.124650 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:14.124595    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:14.296006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:14.295951    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:14.467096 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:14.467009    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:15.737764 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:15.737734    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:16.250599 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:16.250560    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"]
Nov 02 23:26:16.250804 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:16.250779    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb" podUID="c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" containerName="istio-proxy" containerID="cri-o://5092e4f738ec88a88dd8411d47dcfc98a7787e6f0bd865504cc47f08112fd1a6" gracePeriod=30
Nov 02 23:26:16.275743 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:16.275719    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:17.744579 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:17.744545    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:18.434309 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:18.434272    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:18.642247 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:18.642216    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:19.587424 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:19.587383    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:21.553158 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.553133    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:26:21.616406 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.616373    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-token\" (UniqueName: \"kubernetes.io/projected/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-token\") pod \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") "
Nov 02 23:26:21.616545 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.616434    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-envoy\") pod \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") "
Nov 02 23:26:21.616545 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.616466    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-data\") pod \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") "
Nov 02 23:26:21.616545 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.616508    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-workload-socket\") pod \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") "
Nov 02 23:26:21.616719 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.616542    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-kube-api-access-lwq6l\" (UniqueName: \"kubernetes.io/projected/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-kube-api-access-lwq6l\") pod \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") "
Nov 02 23:26:21.616719 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.616586    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istiod-ca-cert\") pod \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") "
Nov 02 23:26:21.616719 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.616640    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-workload-certs\") pod \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") "
Nov 02 23:26:21.616719 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.616659    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-podinfo\") pod \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") "
Nov 02 23:26:21.616719 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.616682    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-credential-socket\") pod \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\" (UID: \"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2\") "
Nov 02 23:26:21.617008 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.616753    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-workload-socket" pod "c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" (UID: "c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:26:21.617008 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.616786    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-data" pod "c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" (UID: "c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:26:21.617008 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.616897    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-workload-certs" pod "c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" (UID: "c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:26:21.617008 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.616999    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-credential-socket" pod "c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" (UID: "c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:26:21.617253 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.617069    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istiod-ca-cert" pod "c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" (UID: "c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:26:21.617432 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.617394    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:26:21.617432 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.617420    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:26:21.617565 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.617434    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:26:21.617565 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.617448    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:26:21.617565 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.617460    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:26:21.618681 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.618653    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-envoy" pod "c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" (UID: "c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:26:21.618857 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.618838    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-token" pod "c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" (UID: "c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:26:21.619389 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.619364    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-podinfo" pod "c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" (UID: "c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:26:21.619540 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.619521    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-kube-api-access-lwq6l" pod "c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" (UID: "c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2"). InnerVolumeSpecName "kube-api-access-lwq6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:26:21.718505 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.718477    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwq6l\" (UniqueName: \"kubernetes.io/projected/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-kube-api-access-lwq6l\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:26:21.718505 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.718505    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:26:21.718676 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.718515    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:26:21.718676 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.718524    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:26:21.827433 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.827352    4294 generic.go:356] "Generic (PLEG): container finished" podID="c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" containerID="5092e4f738ec88a88dd8411d47dcfc98a7787e6f0bd865504cc47f08112fd1a6" exitCode=0
Nov 02 23:26:21.827433 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.827414    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"
Nov 02 23:26:21.827613 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.827431    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb" event={"ID":"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2","Type":"ContainerDied","Data":"5092e4f738ec88a88dd8411d47dcfc98a7787e6f0bd865504cc47f08112fd1a6"}
Nov 02 23:26:21.827613 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.827486    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb" event={"ID":"c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2","Type":"ContainerDied","Data":"b5dbc4804aea02f92da70521bd0f42c443f44a313c1510567b5bfd8721496775"}
Nov 02 23:26:21.827613 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.827508    4294 scope.go:122] "RemoveContainer" containerID="5092e4f738ec88a88dd8411d47dcfc98a7787e6f0bd865504cc47f08112fd1a6"
Nov 02 23:26:21.837449 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.837432    4294 scope.go:122] "RemoveContainer" containerID="39e32cfa3308105a603e7508711f246bc3206b5a8d787e290bf5efe8d27f176e"
Nov 02 23:26:21.848568 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.848543    4294 scope.go:122] "RemoveContainer" containerID="5092e4f738ec88a88dd8411d47dcfc98a7787e6f0bd865504cc47f08112fd1a6"
Nov 02 23:26:21.849054 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:26:21.849029    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5092e4f738ec88a88dd8411d47dcfc98a7787e6f0bd865504cc47f08112fd1a6\": container with ID starting with 5092e4f738ec88a88dd8411d47dcfc98a7787e6f0bd865504cc47f08112fd1a6 not found: ID does not exist" containerID="5092e4f738ec88a88dd8411d47dcfc98a7787e6f0bd865504cc47f08112fd1a6"
Nov 02 23:26:21.849147 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.849061    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5092e4f738ec88a88dd8411d47dcfc98a7787e6f0bd865504cc47f08112fd1a6"} err="failed to get container status \"5092e4f738ec88a88dd8411d47dcfc98a7787e6f0bd865504cc47f08112fd1a6\": rpc error: code = NotFound desc = could not find container \"5092e4f738ec88a88dd8411d47dcfc98a7787e6f0bd865504cc47f08112fd1a6\": container with ID starting with 5092e4f738ec88a88dd8411d47dcfc98a7787e6f0bd865504cc47f08112fd1a6 not found: ID does not exist"
Nov 02 23:26:21.849147 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.849082    4294 scope.go:122] "RemoveContainer" containerID="39e32cfa3308105a603e7508711f246bc3206b5a8d787e290bf5efe8d27f176e"
Nov 02 23:26:21.849360 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:26:21.849338    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39e32cfa3308105a603e7508711f246bc3206b5a8d787e290bf5efe8d27f176e\": container with ID starting with 39e32cfa3308105a603e7508711f246bc3206b5a8d787e290bf5efe8d27f176e not found: ID does not exist" containerID="39e32cfa3308105a603e7508711f246bc3206b5a8d787e290bf5efe8d27f176e"
Nov 02 23:26:21.849406 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.849368    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39e32cfa3308105a603e7508711f246bc3206b5a8d787e290bf5efe8d27f176e"} err="failed to get container status \"39e32cfa3308105a603e7508711f246bc3206b5a8d787e290bf5efe8d27f176e\": rpc error: code = NotFound desc = could not find container \"39e32cfa3308105a603e7508711f246bc3206b5a8d787e290bf5efe8d27f176e\": container with ID starting with 39e32cfa3308105a603e7508711f246bc3206b5a8d787e290bf5efe8d27f176e not found: ID does not exist"
Nov 02 23:26:21.986433 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.986400    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"]
Nov 02 23:26:21.992235 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:21.992202    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9f5790-openshift-default-5bbbdb8597-9xzqb"]
Nov 02 23:26:22.548524 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:22.548486    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" path="/var/lib/kubelet/pods/c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2/volumes"
Nov 02 23:26:22.927268 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:22.927176    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:23.109099 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:23.109064    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:23.279568 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:23.279536    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:25.014341 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:25.014312    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:26.071566 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.071536    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:26.233488 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.233456    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:26.610702 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.610673    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc"]
Nov 02 23:26:26.611228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.611214    4294 cpu_manager.go:414] "containerMap: removing container" podUID="c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" containerName="istio-proxy"
Nov 02 23:26:26.611288 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.611229    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" containerName="istio-proxy"
Nov 02 23:26:26.611288 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.611243    4294 cpu_manager.go:414] "containerMap: removing container" podUID="46b473b0-75fd-4d11-8670-3ffd0abb90ac" containerName="payload-processing"
Nov 02 23:26:26.611288 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.611248    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b473b0-75fd-4d11-8670-3ffd0abb90ac" containerName="payload-processing"
Nov 02 23:26:26.611288 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.611268    4294 cpu_manager.go:414] "containerMap: removing container" podUID="6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e" containerName="payload-processing"
Nov 02 23:26:26.611288 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.611274    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e" containerName="payload-processing"
Nov 02 23:26:26.611288 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.611284    4294 cpu_manager.go:414] "containerMap: removing container" podUID="14520749-ad0f-4518-a370-e5559f3fd565" containerName="payload-processing"
Nov 02 23:26:26.611288 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.611288    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14520749-ad0f-4518-a370-e5559f3fd565" containerName="payload-processing"
Nov 02 23:26:26.611525 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.611299    4294 cpu_manager.go:414] "containerMap: removing container" podUID="8a3538a4-c25e-4d3d-97e5-267a770bbb43" containerName="istio-proxy"
Nov 02 23:26:26.611525 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.611303    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3538a4-c25e-4d3d-97e5-267a770bbb43" containerName="istio-proxy"
Nov 02 23:26:26.611525 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.611371    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b473b0-75fd-4d11-8670-3ffd0abb90ac" containerName="payload-processing"
Nov 02 23:26:26.611525 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.611391    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d2e14f3-dcc6-4e7d-9e41-1ba6e01b972e" containerName="payload-processing"
Nov 02 23:26:26.611525 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.611397    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" containerName="istio-proxy"
Nov 02 23:26:26.611525 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.611403    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="14520749-ad0f-4518-a370-e5559f3fd565" containerName="payload-processing"
Nov 02 23:26:26.611525 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.611410    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a3538a4-c25e-4d3d-97e5-267a770bbb43" containerName="istio-proxy"
Nov 02 23:26:26.614832 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.614815    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc"
Nov 02 23:26:26.666392 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.666367    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc"]
Nov 02 23:26:26.710427 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.710383    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-3bf62f\""
Nov 02 23:26:26.710589 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.710573    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-3bf62f-dockercfg-h2j5g\""
Nov 02 23:26:26.782665 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.782627    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc\" (UID: \"0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc"
Nov 02 23:26:26.782840 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.782676    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m7tb\" (UniqueName: \"kubernetes.io/projected/0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d-kube-api-access-6m7tb\") pod \"payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc\" (UID: \"0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc"
Nov 02 23:26:26.883963 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.883876    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc\" (UID: \"0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc"
Nov 02 23:26:26.883963 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.883920    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m7tb\" (UniqueName: \"kubernetes.io/projected/0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d-kube-api-access-6m7tb\") pod \"payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc\" (UID: \"0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc"
Nov 02 23:26:26.884592 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.884571    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc\" (UID: \"0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc"
Nov 02 23:26:26.939587 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:26.939554    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m7tb\" (UniqueName: \"kubernetes.io/projected/0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d-kube-api-access-6m7tb\") pod \"payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc\" (UID: \"0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc"
Nov 02 23:26:27.229439 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:27.229353    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc"
Nov 02 23:26:27.593664 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:27.593636    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc"]
Nov 02 23:26:27.594149 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:26:27.594117    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb1a960_15cf_492e_90a4_eb2cd0fb1b6d.slice/crio-4cff90053b48d676c47a29de246533d6c564447d07656d35f5df6be0fb2eb6cb WatchSource:0}: Error finding container 4cff90053b48d676c47a29de246533d6c564447d07656d35f5df6be0fb2eb6cb: Status 404 returned error can't find the container with id 4cff90053b48d676c47a29de246533d6c564447d07656d35f5df6be0fb2eb6cb
Nov 02 23:26:27.865802 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:27.865714    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc" event={"ID":"0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d","Type":"ContainerStarted","Data":"784ad7f9a4f08aee1cea128411f7b776dddfcf89b053caf969fcfb9bc929d701"}
Nov 02 23:26:27.865802 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:27.865772    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc" event={"ID":"0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d","Type":"ContainerStarted","Data":"4cff90053b48d676c47a29de246533d6c564447d07656d35f5df6be0fb2eb6cb"}
Nov 02 23:26:27.865802 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:27.865801    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc"
Nov 02 23:26:27.977288 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:27.977233    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc" podStartSLOduration=1.977218184 podStartE2EDuration="1.977218184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:26:27.975170404 +0000 UTC m=+2027.543525192" watchObservedRunningTime="2026-11-02 23:26:27.977218184 +0000 UTC m=+2027.545572970"
Nov 02 23:26:30.998717 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:30.998677    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:32.892269 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:32.892233    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:34.922708 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:34.922675    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:35.460701 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:35.460653    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:38.287621 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:38.287589    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:38.467430 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:38.467393    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:38.637665 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:38.637584    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:38.874258 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:38.874225    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc"
Nov 02 23:26:41.757386 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:26:41.757348    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:26:43.637628 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:43.637595    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:48.430648 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:48.430612    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:53.225869 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:53.225834    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:53.382689 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:53.382653    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:53.771505 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:53.771476    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:54.081790 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:54.081707    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:58.486087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:58.486036    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:58.654099 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:58.654066    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:26:58.827554 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:26:58.827520    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:01.629293 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:01.629262    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:01.800220 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:01.800190    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:05.822517 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:05.822484    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:10.865746 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:10.865713    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:11.751481 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:27:11.751445    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:27:15.842342 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:15.842303    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:18.039226 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:18.039188    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:21.079856 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:21.079818    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:21.249586 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:21.249553    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:23.695652 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:23.695618    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:24.198105 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:24.198071    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"]
Nov 02 23:27:24.198306 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:24.198271    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24" podUID="85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" containerName="istio-proxy" containerID="cri-o://3047e779ffd360fdf29a143952d9fe45dd30422e7dd7434fe692b6b4776af145" gracePeriod=30
Nov 02 23:27:25.857223 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:25.857187    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sfd24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 23:27:25.857701 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:25.857254    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24" podUID="85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 23:27:26.067594 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:26.067559    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:29.255740 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.255708    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:29.472848 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.472776    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:27:29.640777 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.640738    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-podinfo\") pod \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") "
Nov 02 23:27:29.641018 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.640788    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-workload-certs\") pod \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") "
Nov 02 23:27:29.641018 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.640821    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istiod-ca-cert\") pod \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") "
Nov 02 23:27:29.641018 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.640897    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-data\") pod \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") "
Nov 02 23:27:29.641018 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.640936    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-token\" (UniqueName: \"kubernetes.io/projected/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-token\") pod \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") "
Nov 02 23:27:29.641018 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.641001    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-kube-api-access-vfhgn\" (UniqueName: \"kubernetes.io/projected/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-kube-api-access-vfhgn\") pod \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") "
Nov 02 23:27:29.641305 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.641122    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-credential-socket\") pod \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") "
Nov 02 23:27:29.641305 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.641198    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-envoy\") pod \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") "
Nov 02 23:27:29.641305 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.641281    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-workload-socket\") pod \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\" (UID: \"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7\") "
Nov 02 23:27:29.641461 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.641370    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-data" pod "85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" (UID: "85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:27:29.641622 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.641594    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istiod-ca-cert" pod "85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" (UID: "85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:27:29.642229 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.641799    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-credential-socket" pod "85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" (UID: "85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:27:29.642229 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.641823    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:27:29.642229 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.641844    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:27:29.642229 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.642188    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-workload-socket" pod "85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" (UID: "85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:27:29.642549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.642250    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-workload-certs" pod "85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" (UID: "85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:27:29.643853 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.643820    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-podinfo" pod "85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" (UID: "85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:27:29.644014 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.643886    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-kube-api-access-vfhgn" pod "85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" (UID: "85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7"). InnerVolumeSpecName "kube-api-access-vfhgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:27:29.644014 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.643939    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-envoy" pod "85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" (UID: "85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:27:29.644141 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.644068    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-token" pod "85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" (UID: "85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:27:29.745504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.745383    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfhgn\" (UniqueName: \"kubernetes.io/projected/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-kube-api-access-vfhgn\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:27:29.745504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.745422    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:27:29.745504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.745453    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:27:29.745504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.745467    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:27:29.745504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.745481    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:27:29.745504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.745495    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:27:29.745504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:29.745508    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:27:30.178201 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:30.178106    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:30.249722 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:30.249687    4294 generic.go:356] "Generic (PLEG): container finished" podID="85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" containerID="3047e779ffd360fdf29a143952d9fe45dd30422e7dd7434fe692b6b4776af145" exitCode=0
Nov 02 23:27:30.249924 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:30.249790    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24" event={"ID":"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7","Type":"ContainerDied","Data":"3047e779ffd360fdf29a143952d9fe45dd30422e7dd7434fe692b6b4776af145"}
Nov 02 23:27:30.249924 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:30.249800    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"
Nov 02 23:27:30.249924 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:30.249843    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24" event={"ID":"85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7","Type":"ContainerDied","Data":"c0c1ebec4ea2b9eab4135c6aa17ea384ca74e47ca0a38bc6a149151807e1c178"}
Nov 02 23:27:30.249924 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:30.249872    4294 scope.go:122] "RemoveContainer" containerID="3047e779ffd360fdf29a143952d9fe45dd30422e7dd7434fe692b6b4776af145"
Nov 02 23:27:30.262050 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:30.262028    4294 scope.go:122] "RemoveContainer" containerID="3047e779ffd360fdf29a143952d9fe45dd30422e7dd7434fe692b6b4776af145"
Nov 02 23:27:30.262436 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:27:30.262415    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3047e779ffd360fdf29a143952d9fe45dd30422e7dd7434fe692b6b4776af145\": container with ID starting with 3047e779ffd360fdf29a143952d9fe45dd30422e7dd7434fe692b6b4776af145 not found: ID does not exist" containerID="3047e779ffd360fdf29a143952d9fe45dd30422e7dd7434fe692b6b4776af145"
Nov 02 23:27:30.262481 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:30.262445    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3047e779ffd360fdf29a143952d9fe45dd30422e7dd7434fe692b6b4776af145"} err="failed to get container status \"3047e779ffd360fdf29a143952d9fe45dd30422e7dd7434fe692b6b4776af145\": rpc error: code = NotFound desc = could not find container \"3047e779ffd360fdf29a143952d9fe45dd30422e7dd7434fe692b6b4776af145\": container with ID starting with 3047e779ffd360fdf29a143952d9fe45dd30422e7dd7434fe692b6b4776af145 not found: ID does not exist"
Nov 02 23:27:30.342811 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:30.342777    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:30.409257 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:30.409218    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"]
Nov 02 23:27:30.455094 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:30.455015    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sfd24"]
Nov 02 23:27:30.547561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:30.547532    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" path="/var/lib/kubelet/pods/85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7/volumes"
Nov 02 23:27:31.081901 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.081869    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"]
Nov 02 23:27:31.082445 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.082432    4294 cpu_manager.go:414] "containerMap: removing container" podUID="85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" containerName="istio-proxy"
Nov 02 23:27:31.082508 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.082447    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" containerName="istio-proxy"
Nov 02 23:27:31.082508 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.082470    4294 cpu_manager.go:414] "containerMap: removing container" podUID="c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" containerName="istio-proxy"
Nov 02 23:27:31.082508 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.082475    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" containerName="istio-proxy"
Nov 02 23:27:31.082600 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.082565    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9daec5e-f7ee-4d62-be3f-3d6a3ef1b9e2" containerName="istio-proxy"
Nov 02 23:27:31.082600 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.082575    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="85cbb9b4-2bbe-41c6-95eb-af57b5ea02b7" containerName="istio-proxy"
Nov 02 23:27:31.087434 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.087415    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"
Nov 02 23:27:31.156888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.147788    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"]
Nov 02 23:27:31.173773 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.173721    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44063d97-ef69-410b-af13-2c7896689fc3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4\" (UID: \"44063d97-ef69-410b-af13-2c7896689fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"
Nov 02 23:27:31.174015 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.173831    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44063d97-ef69-410b-af13-2c7896689fc3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4\" (UID: \"44063d97-ef69-410b-af13-2c7896689fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"
Nov 02 23:27:31.174015 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.173972    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr6bw\" (UniqueName: \"kubernetes.io/projected/44063d97-ef69-410b-af13-2c7896689fc3-kube-api-access-gr6bw\") pod \"maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4\" (UID: \"44063d97-ef69-410b-af13-2c7896689fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"
Nov 02 23:27:31.174154 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.174029    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44063d97-ef69-410b-af13-2c7896689fc3-metrics-certs\") pod \"maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4\" (UID: \"44063d97-ef69-410b-af13-2c7896689fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"
Nov 02 23:27:31.191027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.190995    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-3bf62f-serving-cert\""
Nov 02 23:27:31.275149 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.275105    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr6bw\" (UniqueName: \"kubernetes.io/projected/44063d97-ef69-410b-af13-2c7896689fc3-kube-api-access-gr6bw\") pod \"maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4\" (UID: \"44063d97-ef69-410b-af13-2c7896689fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"
Nov 02 23:27:31.275632 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.275170    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44063d97-ef69-410b-af13-2c7896689fc3-metrics-certs\") pod \"maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4\" (UID: \"44063d97-ef69-410b-af13-2c7896689fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"
Nov 02 23:27:31.275632 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.275263    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44063d97-ef69-410b-af13-2c7896689fc3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4\" (UID: \"44063d97-ef69-410b-af13-2c7896689fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"
Nov 02 23:27:31.275632 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.275343    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44063d97-ef69-410b-af13-2c7896689fc3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4\" (UID: \"44063d97-ef69-410b-af13-2c7896689fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"
Nov 02 23:27:31.275632 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:27:31.275417    4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-serving-cert: secret "maas-api-e2e-praxis-a-3bf62f-serving-cert" not found
Nov 02 23:27:31.275632 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:27:31.275501    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44063d97-ef69-410b-af13-2c7896689fc3-maas-api-tls podName:44063d97-ef69-410b-af13-2c7896689fc3 nodeName:}" failed. No retries permitted until 2026-11-02 23:27:31.775478653 +0000 UTC m=+2091.343833419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/44063d97-ef69-410b-af13-2c7896689fc3-maas-api-tls") pod "maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4" (UID: "44063d97-ef69-410b-af13-2c7896689fc3") : secret "maas-api-e2e-praxis-a-3bf62f-serving-cert" not found
Nov 02 23:27:31.275970 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.275950    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44063d97-ef69-410b-af13-2c7896689fc3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4\" (UID: \"44063d97-ef69-410b-af13-2c7896689fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"
Nov 02 23:27:31.277681 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.277661    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44063d97-ef69-410b-af13-2c7896689fc3-metrics-certs\") pod \"maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4\" (UID: \"44063d97-ef69-410b-af13-2c7896689fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"
Nov 02 23:27:31.332157 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.332076    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr6bw\" (UniqueName: \"kubernetes.io/projected/44063d97-ef69-410b-af13-2c7896689fc3-kube-api-access-gr6bw\") pod \"maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4\" (UID: \"44063d97-ef69-410b-af13-2c7896689fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"
Nov 02 23:27:31.334479 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.334448    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:31.506830 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.506775    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:31.656711 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.656620    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:31.781552 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.781513    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44063d97-ef69-410b-af13-2c7896689fc3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4\" (UID: \"44063d97-ef69-410b-af13-2c7896689fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"
Nov 02 23:27:31.784298 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:31.784270    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44063d97-ef69-410b-af13-2c7896689fc3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4\" (UID: \"44063d97-ef69-410b-af13-2c7896689fc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"
Nov 02 23:27:32.000975 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:32.000939    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"
Nov 02 23:27:32.369579 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:32.369551    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"]
Nov 02 23:27:32.370132 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:27:32.370101    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44063d97_ef69_410b_af13_2c7896689fc3.slice/crio-932ba42938bf4c6c1cc6a7877b304136a69262ab44b300952d1eb6e6f13cf522 WatchSource:0}: Error finding container 932ba42938bf4c6c1cc6a7877b304136a69262ab44b300952d1eb6e6f13cf522: Status 404 returned error can't find the container with id 932ba42938bf4c6c1cc6a7877b304136a69262ab44b300952d1eb6e6f13cf522
Nov 02 23:27:32.371700 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:32.371676    4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 23:27:33.134310 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:33.134273    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:33.277044 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:33.277004    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4" event={"ID":"44063d97-ef69-410b-af13-2c7896689fc3","Type":"ContainerStarted","Data":"932ba42938bf4c6c1cc6a7877b304136a69262ab44b300952d1eb6e6f13cf522"}
Nov 02 23:27:33.300129 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:33.300090    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:33.457784 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:33.457724    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:34.285438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:34.285397    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4" event={"ID":"44063d97-ef69-410b-af13-2c7896689fc3","Type":"ContainerStarted","Data":"97f063b9b22acd81a4b31fdd2cdec1bb43dc68ac1827d3dfc797a43eb92dcd1c"}
Nov 02 23:27:34.285637 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:34.285465    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"
Nov 02 23:27:34.398224 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:34.398167    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4" podStartSLOduration=2.4010436090000002 podStartE2EDuration="3.398150747s" totalImagesPullingTime="997.107138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:27:34.395228746 +0000 UTC m=+2093.963583544" watchObservedRunningTime="2026-11-02 23:27:34.398150747 +0000 UTC m=+2093.966505533"
Nov 02 23:27:34.831497 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:34.831459    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:36.479840 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:36.479807    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:36.846767 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:36.846736    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:37.020342 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:37.020308    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:38.823253 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:38.823193    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:40.296343 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:40.296312    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"
Nov 02 23:27:41.754467 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:27:41.754432    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:27:41.870622 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:41.870591    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:48.693471 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:48.693438    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:53.583609 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:53.583574    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:57.599031 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:57.598993    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:58.656975 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:58.656942    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:58.822119 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:58.822088    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:59.245247 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:59.245217    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:27:59.629510 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:27:59.629433    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:03.158789 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:03.158750    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:03.165060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:03.165032    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"]
Nov 02 23:28:03.165288 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:03.165266    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9" podUID="300f6dfb-aeb3-4967-859b-664122971d83" containerName="maas-api" containerID="cri-o://df8c1e06d3bdd0198f5577e5d41e7c80ac95751ca0dfc89730ea6692335451f1" gracePeriod=30
Nov 02 23:28:03.334991 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:03.334946    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:03.460161 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:03.460133    4294 generic.go:356] "Generic (PLEG): container finished" podID="300f6dfb-aeb3-4967-859b-664122971d83" containerID="df8c1e06d3bdd0198f5577e5d41e7c80ac95751ca0dfc89730ea6692335451f1" exitCode=0
Nov 02 23:28:03.460281 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:03.460212    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9" event={"ID":"300f6dfb-aeb3-4967-859b-664122971d83","Type":"ContainerDied","Data":"df8c1e06d3bdd0198f5577e5d41e7c80ac95751ca0dfc89730ea6692335451f1"}
Nov 02 23:28:03.474132 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:03.474108    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"
Nov 02 23:28:03.529845 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:03.529811    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:03.588383 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:03.588352    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/300f6dfb-aeb3-4967-859b-664122971d83-kube-api-access-vhxzj\" (UniqueName: \"kubernetes.io/projected/300f6dfb-aeb3-4967-859b-664122971d83-kube-api-access-vhxzj\") pod \"300f6dfb-aeb3-4967-859b-664122971d83\" (UID: \"300f6dfb-aeb3-4967-859b-664122971d83\") "
Nov 02 23:28:03.588578 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:03.588447    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/300f6dfb-aeb3-4967-859b-664122971d83-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/300f6dfb-aeb3-4967-859b-664122971d83-maas-api-tls\") pod \"300f6dfb-aeb3-4967-859b-664122971d83\" (UID: \"300f6dfb-aeb3-4967-859b-664122971d83\") "
Nov 02 23:28:03.588632 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:03.588613    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/300f6dfb-aeb3-4967-859b-664122971d83-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/300f6dfb-aeb3-4967-859b-664122971d83-openshift-service-ca\") pod \"300f6dfb-aeb3-4967-859b-664122971d83\" (UID: \"300f6dfb-aeb3-4967-859b-664122971d83\") "
Nov 02 23:28:03.588672 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:03.588629    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/300f6dfb-aeb3-4967-859b-664122971d83-metrics-certs\" (UniqueName: \"kubernetes.io/secret/300f6dfb-aeb3-4967-859b-664122971d83-metrics-certs\") pod \"300f6dfb-aeb3-4967-859b-664122971d83\" (UID: \"300f6dfb-aeb3-4967-859b-664122971d83\") "
Nov 02 23:28:03.589116 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:03.589083    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/300f6dfb-aeb3-4967-859b-664122971d83-openshift-service-ca" pod "300f6dfb-aeb3-4967-859b-664122971d83" (UID: "300f6dfb-aeb3-4967-859b-664122971d83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:28:03.591569 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:03.591543    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300f6dfb-aeb3-4967-859b-664122971d83-kube-api-access-vhxzj" pod "300f6dfb-aeb3-4967-859b-664122971d83" (UID: "300f6dfb-aeb3-4967-859b-664122971d83"). InnerVolumeSpecName "kube-api-access-vhxzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:28:03.607787 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:03.607683    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/300f6dfb-aeb3-4967-859b-664122971d83-metrics-certs" pod "300f6dfb-aeb3-4967-859b-664122971d83" (UID: "300f6dfb-aeb3-4967-859b-664122971d83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:28:03.608292 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:03.608147    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/300f6dfb-aeb3-4967-859b-664122971d83-maas-api-tls" pod "300f6dfb-aeb3-4967-859b-664122971d83" (UID: "300f6dfb-aeb3-4967-859b-664122971d83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:28:03.690580 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:03.690553    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/300f6dfb-aeb3-4967-859b-664122971d83-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:28:03.690580 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:03.690579    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/300f6dfb-aeb3-4967-859b-664122971d83-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:28:03.690800 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:03.690589    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhxzj\" (UniqueName: \"kubernetes.io/projected/300f6dfb-aeb3-4967-859b-664122971d83-kube-api-access-vhxzj\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:28:03.690800 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:03.690599    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/300f6dfb-aeb3-4967-859b-664122971d83-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:28:04.466580 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:04.466547    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"
Nov 02 23:28:04.467053 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:04.466576    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9" event={"ID":"300f6dfb-aeb3-4967-859b-664122971d83","Type":"ContainerDied","Data":"3dd542a483fa09fe0b66053e5737cc99dff252bdc83b8f591b6b114b4be99466"}
Nov 02 23:28:04.467053 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:04.466643    4294 scope.go:122] "RemoveContainer" containerID="df8c1e06d3bdd0198f5577e5d41e7c80ac95751ca0dfc89730ea6692335451f1"
Nov 02 23:28:04.626204 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:04.626176    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"]
Nov 02 23:28:04.673858 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:04.673827    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b57963-67cf9c4b44-4mmp9"]
Nov 02 23:28:04.894067 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:04.894034    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"]
Nov 02 23:28:04.894670 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:04.894650    4294 cpu_manager.go:414] "containerMap: removing container" podUID="300f6dfb-aeb3-4967-859b-664122971d83" containerName="maas-api"
Nov 02 23:28:04.894670 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:04.894671    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="300f6dfb-aeb3-4967-859b-664122971d83" containerName="maas-api"
Nov 02 23:28:04.894851 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:04.894798    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="300f6dfb-aeb3-4967-859b-664122971d83" containerName="maas-api"
Nov 02 23:28:04.899240 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:04.899219    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"
Nov 02 23:28:04.952915 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:04.952889    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"]
Nov 02 23:28:05.005616 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:05.005588    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2\" (UID: \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"
Nov 02 23:28:05.005762 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:05.005639    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf9t4\" (UniqueName: \"kubernetes.io/projected/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-kube-api-access-nf9t4\") pod \"maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2\" (UID: \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"
Nov 02 23:28:05.005824 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:05.005803    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2\" (UID: \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"
Nov 02 23:28:05.005871 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:05.005855    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-metrics-certs\") pod \"maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2\" (UID: \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"
Nov 02 23:28:05.042821 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:05.042796    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5ff336-serving-cert\""
Nov 02 23:28:05.107142 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:05.107113    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2\" (UID: \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"
Nov 02 23:28:05.107314 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:05.107159    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf9t4\" (UniqueName: \"kubernetes.io/projected/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-kube-api-access-nf9t4\") pod \"maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2\" (UID: \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"
Nov 02 23:28:05.107314 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:05.107246    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2\" (UID: \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"
Nov 02 23:28:05.107314 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:28:05.107252    4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-serving-cert: secret "maas-api-e2e-ipp-b-5ff336-serving-cert" not found
Nov 02 23:28:05.107314 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:05.107289    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-metrics-certs\") pod \"maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2\" (UID: \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"
Nov 02 23:28:05.107520 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:28:05.107337    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-maas-api-tls podName:b91d5fc6-8e5d-44e9-be6d-58822e9697bb nodeName:}" failed. No retries permitted until 2026-11-02 23:28:05.607314992 +0000 UTC m=+2125.175669774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-maas-api-tls") pod "maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2" (UID: "b91d5fc6-8e5d-44e9-be6d-58822e9697bb") : secret "maas-api-e2e-ipp-b-5ff336-serving-cert" not found
Nov 02 23:28:05.107849 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:05.107827    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2\" (UID: \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"
Nov 02 23:28:05.109738 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:05.109714    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-metrics-certs\") pod \"maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2\" (UID: \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"
Nov 02 23:28:05.163216 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:05.163147    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf9t4\" (UniqueName: \"kubernetes.io/projected/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-kube-api-access-nf9t4\") pod \"maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2\" (UID: \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"
Nov 02 23:28:05.175004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:05.174958    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:05.333474 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:05.333439    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:05.505437 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:05.505405    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:05.614381 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:05.614348    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2\" (UID: \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"
Nov 02 23:28:05.617050 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:05.617025    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2\" (UID: \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"
Nov 02 23:28:05.812697 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:05.812614    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"
Nov 02 23:28:06.182730 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:06.182702    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"]
Nov 02 23:28:06.183073 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:28:06.183047    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91d5fc6_8e5d_44e9_be6d_58822e9697bb.slice/crio-30f7bf51774d17f0eb5790d375522a2e19f7d8f64bc5c9d76afe312c5c0ce4ad WatchSource:0}: Error finding container 30f7bf51774d17f0eb5790d375522a2e19f7d8f64bc5c9d76afe312c5c0ce4ad: Status 404 returned error can't find the container with id 30f7bf51774d17f0eb5790d375522a2e19f7d8f64bc5c9d76afe312c5c0ce4ad
Nov 02 23:28:06.480039 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:06.479936    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2" event={"ID":"b91d5fc6-8e5d-44e9-be6d-58822e9697bb","Type":"ContainerStarted","Data":"30f7bf51774d17f0eb5790d375522a2e19f7d8f64bc5c9d76afe312c5c0ce4ad"}
Nov 02 23:28:06.547188 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:06.547160    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="300f6dfb-aeb3-4967-859b-664122971d83" path="/var/lib/kubelet/pods/300f6dfb-aeb3-4967-859b-664122971d83/volumes"
Nov 02 23:28:06.661212 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:06.661176    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:06.831172 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:06.831139    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:07.486753 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:07.486657    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2" event={"ID":"b91d5fc6-8e5d-44e9-be6d-58822e9697bb","Type":"ContainerStarted","Data":"32d13b54a9e71044a754f0d440242e178c3a4373a8bfa648e97628a790528072"}
Nov 02 23:28:07.486753 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:07.486746    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"
Nov 02 23:28:07.599453 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:07.599390    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2" podStartSLOduration=2.7058342829999997 podStartE2EDuration="3.599376875s" totalImagesPullingTime="893.542592ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:28:07.596088354 +0000 UTC m=+2127.164443141" watchObservedRunningTime="2026-11-02 23:28:07.599376875 +0000 UTC m=+2127.167731693"
Nov 02 23:28:08.041664 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:08.041625    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:08.217152 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:08.217116    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:09.013474 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:09.013439    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:09.588043 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:09.588007    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:10.169806 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:10.169777    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:10.342992 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:10.342951    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:10.812681 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:10.812641    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:10.977609 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:10.977580    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:11.750798 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:28:11.750762    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:28:12.313005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:12.312951    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:13.503411 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:13.503372    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"
Nov 02 23:28:15.917758 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:15.917690    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:16.081888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:16.081849    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:16.341603 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:16.341570    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:17.014227 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.014188    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"]
Nov 02 23:28:17.014698 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.014434    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6" podUID="28a17c69-bf9c-4920-b804-429af4fbfadf" containerName="discovery" containerID="cri-o://f76f5596d63b6c425ec113d94906be1680a9f5547cf6f7bec35345b0561f9d5b" gracePeriod=30
Nov 02 23:28:17.074521 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.074493    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:17.300127 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.300103    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:28:17.466428 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.466393    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-kubeconfig\") pod \"28a17c69-bf9c-4920-b804-429af4fbfadf\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") "
Nov 02 23:28:17.466615 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.466502    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-cacerts\" (UniqueName: \"kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-cacerts\") pod \"28a17c69-bf9c-4920-b804-429af4fbfadf\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") "
Nov 02 23:28:17.466615 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.466529    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-csr-ca-configmap\") pod \"28a17c69-bf9c-4920-b804-429af4fbfadf\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") "
Nov 02 23:28:17.466615 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.466549    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-csr-dns-cert\") pod \"28a17c69-bf9c-4920-b804-429af4fbfadf\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") "
Nov 02 23:28:17.466797 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.466649    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a17c69-bf9c-4920-b804-429af4fbfadf-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a17c69-bf9c-4920-b804-429af4fbfadf-local-certs\") pod \"28a17c69-bf9c-4920-b804-429af4fbfadf\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") "
Nov 02 23:28:17.466797 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.466710    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-token\" (UniqueName: \"kubernetes.io/projected/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-token\") pod \"28a17c69-bf9c-4920-b804-429af4fbfadf\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") "
Nov 02 23:28:17.466797 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.466738    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a17c69-bf9c-4920-b804-429af4fbfadf-kube-api-access-t5k2n\" (UniqueName: \"kubernetes.io/projected/28a17c69-bf9c-4920-b804-429af4fbfadf-kube-api-access-t5k2n\") pod \"28a17c69-bf9c-4920-b804-429af4fbfadf\" (UID: \"28a17c69-bf9c-4920-b804-429af4fbfadf\") "
Nov 02 23:28:17.466965 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.466930    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-csr-ca-configmap" pod "28a17c69-bf9c-4920-b804-429af4fbfadf" (UID: "28a17c69-bf9c-4920-b804-429af4fbfadf"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:28:17.467337 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.467286    4294 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-csr-ca-configmap\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:28:17.469004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.468949    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-kubeconfig" pod "28a17c69-bf9c-4920-b804-429af4fbfadf" (UID: "28a17c69-bf9c-4920-b804-429af4fbfadf"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:28:17.469214 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.469196    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a17c69-bf9c-4920-b804-429af4fbfadf-kube-api-access-t5k2n" pod "28a17c69-bf9c-4920-b804-429af4fbfadf" (UID: "28a17c69-bf9c-4920-b804-429af4fbfadf"). InnerVolumeSpecName "kube-api-access-t5k2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:28:17.469296 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.469271    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-cacerts" pod "28a17c69-bf9c-4920-b804-429af4fbfadf" (UID: "28a17c69-bf9c-4920-b804-429af4fbfadf"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:28:17.469427 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.469401    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-token" pod "28a17c69-bf9c-4920-b804-429af4fbfadf" (UID: "28a17c69-bf9c-4920-b804-429af4fbfadf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:28:17.469427 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.469421    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-csr-dns-cert" pod "28a17c69-bf9c-4920-b804-429af4fbfadf" (UID: "28a17c69-bf9c-4920-b804-429af4fbfadf"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:28:17.469559 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.469426    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a17c69-bf9c-4920-b804-429af4fbfadf-local-certs" pod "28a17c69-bf9c-4920-b804-429af4fbfadf" (UID: "28a17c69-bf9c-4920-b804-429af4fbfadf"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:28:17.555469 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.555371    4294 generic.go:356] "Generic (PLEG): container finished" podID="28a17c69-bf9c-4920-b804-429af4fbfadf" containerID="f76f5596d63b6c425ec113d94906be1680a9f5547cf6f7bec35345b0561f9d5b" exitCode=0
Nov 02 23:28:17.555469 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.555420    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"
Nov 02 23:28:17.555469 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.555443    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6" event={"ID":"28a17c69-bf9c-4920-b804-429af4fbfadf","Type":"ContainerDied","Data":"f76f5596d63b6c425ec113d94906be1680a9f5547cf6f7bec35345b0561f9d5b"}
Nov 02 23:28:17.555715 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.555489    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6" event={"ID":"28a17c69-bf9c-4920-b804-429af4fbfadf","Type":"ContainerDied","Data":"6d0c1b56fd41b414399f5a8ff883fb73001d2ac479f391f8e9488250ff63a933"}
Nov 02 23:28:17.555715 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.555508    4294 scope.go:122] "RemoveContainer" containerID="f76f5596d63b6c425ec113d94906be1680a9f5547cf6f7bec35345b0561f9d5b"
Nov 02 23:28:17.565534 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.565517    4294 scope.go:122] "RemoveContainer" containerID="f76f5596d63b6c425ec113d94906be1680a9f5547cf6f7bec35345b0561f9d5b"
Nov 02 23:28:17.565773 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:28:17.565756    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76f5596d63b6c425ec113d94906be1680a9f5547cf6f7bec35345b0561f9d5b\": container with ID starting with f76f5596d63b6c425ec113d94906be1680a9f5547cf6f7bec35345b0561f9d5b not found: ID does not exist" containerID="f76f5596d63b6c425ec113d94906be1680a9f5547cf6f7bec35345b0561f9d5b"
Nov 02 23:28:17.565843 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.565779    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76f5596d63b6c425ec113d94906be1680a9f5547cf6f7bec35345b0561f9d5b"} err="failed to get container status \"f76f5596d63b6c425ec113d94906be1680a9f5547cf6f7bec35345b0561f9d5b\": rpc error: code = NotFound desc = could not find container \"f76f5596d63b6c425ec113d94906be1680a9f5547cf6f7bec35345b0561f9d5b\": container with ID starting with f76f5596d63b6c425ec113d94906be1680a9f5547cf6f7bec35345b0561f9d5b not found: ID does not exist"
Nov 02 23:28:17.567967 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.567941    4294 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28a17c69-bf9c-4920-b804-429af4fbfadf-local-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:28:17.568114 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.567970    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:28:17.568114 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.568011    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5k2n\" (UniqueName: \"kubernetes.io/projected/28a17c69-bf9c-4920-b804-429af4fbfadf-kube-api-access-t5k2n\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:28:17.568114 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.568019    4294 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-kubeconfig\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:28:17.568114 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.568029    4294 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-cacerts\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:28:17.568114 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.568038    4294 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28a17c69-bf9c-4920-b804-429af4fbfadf-istio-csr-dns-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:28:17.721194 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.721158    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"]
Nov 02 23:28:17.724599 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:17.724572    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hmbx6"]
Nov 02 23:28:18.547663 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:18.547631    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a17c69-bf9c-4920-b804-429af4fbfadf" path="/var/lib/kubelet/pods/28a17c69-bf9c-4920-b804-429af4fbfadf/volumes"
Nov 02 23:28:18.916421 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:18.916345    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:20.121638 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:20.121602    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:20.415761 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:20.415673    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:21.607761 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:21.607733    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:22.130452 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:22.130418    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:22.296779 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:22.296744    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:22.834772 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:22.834739    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:24.725239 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:24.725194    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:28.608147 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:28.608108    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:28.832166 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:28.832131    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:29.142924 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:29.142890    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:32.220382 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:32.220349    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:33.558818 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:33.558786    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:33.713544 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:33.713510    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"]
Nov 02 23:28:33.713754 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:33.713728    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws" podUID="72b6e169-e4d8-4a5d-b5a2-88d4c424503e" containerName="istio-proxy" containerID="cri-o://15c419604dc515d6a1bc875013b3abd7fdfd89642fc82d30be32aa48ca3aaa22" gracePeriod=30
Nov 02 23:28:35.305273 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:35.305239    4294 patch_prober.go:29] interesting pod/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 23:28:35.305766 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:35.305294    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws" podUID="72b6e169-e4d8-4a5d-b5a2-88d4c424503e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 23:28:37.463925 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:37.463892    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:38.984727 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:38.984674    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:28:39.107074 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.107041    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-workload-socket\") pod \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") "
Nov 02 23:28:39.107265 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.107094    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-kube-api-access-wlh45\" (UniqueName: \"kubernetes.io/projected/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-kube-api-access-wlh45\") pod \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") "
Nov 02 23:28:39.107265 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.107125    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-workload-certs\") pod \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") "
Nov 02 23:28:39.107265 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.107148    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-envoy\") pod \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") "
Nov 02 23:28:39.107449 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.107283    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-podinfo\") pod \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") "
Nov 02 23:28:39.107449 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.107322    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-data\") pod \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") "
Nov 02 23:28:39.107449 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.107352    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istiod-ca-cert\") pod \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") "
Nov 02 23:28:39.107449 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.107388    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-workload-socket" pod "72b6e169-e4d8-4a5d-b5a2-88d4c424503e" (UID: "72b6e169-e4d8-4a5d-b5a2-88d4c424503e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:28:39.107672 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.107462    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-token\" (UniqueName: \"kubernetes.io/projected/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-token\") pod \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") "
Nov 02 23:28:39.107672 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.107496    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-credential-socket\") pod \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\" (UID: \"72b6e169-e4d8-4a5d-b5a2-88d4c424503e\") "
Nov 02 23:28:39.107811 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.107788    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istiod-ca-cert" pod "72b6e169-e4d8-4a5d-b5a2-88d4c424503e" (UID: "72b6e169-e4d8-4a5d-b5a2-88d4c424503e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:28:39.107912 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.107634    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-data" pod "72b6e169-e4d8-4a5d-b5a2-88d4c424503e" (UID: "72b6e169-e4d8-4a5d-b5a2-88d4c424503e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:28:39.107993 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.107932    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-credential-socket" pod "72b6e169-e4d8-4a5d-b5a2-88d4c424503e" (UID: "72b6e169-e4d8-4a5d-b5a2-88d4c424503e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:28:39.108082 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.108061    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-workload-certs" pod "72b6e169-e4d8-4a5d-b5a2-88d4c424503e" (UID: "72b6e169-e4d8-4a5d-b5a2-88d4c424503e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:28:39.108183 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.108163    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:28:39.108233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.108188    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:28:39.108233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.108204    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:28:39.108233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.108219    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:28:39.108324 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.108233    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:28:39.110050 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.110014    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-podinfo" pod "72b6e169-e4d8-4a5d-b5a2-88d4c424503e" (UID: "72b6e169-e4d8-4a5d-b5a2-88d4c424503e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:28:39.110050 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.110045    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-token" pod "72b6e169-e4d8-4a5d-b5a2-88d4c424503e" (UID: "72b6e169-e4d8-4a5d-b5a2-88d4c424503e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:28:39.110208 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.110058    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-envoy" pod "72b6e169-e4d8-4a5d-b5a2-88d4c424503e" (UID: "72b6e169-e4d8-4a5d-b5a2-88d4c424503e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:28:39.110208 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.110154    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-kube-api-access-wlh45" pod "72b6e169-e4d8-4a5d-b5a2-88d4c424503e" (UID: "72b6e169-e4d8-4a5d-b5a2-88d4c424503e"). InnerVolumeSpecName "kube-api-access-wlh45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:28:39.209612 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.209576    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:28:39.209612 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.209610    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:28:39.209825 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.209626    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlh45\" (UniqueName: \"kubernetes.io/projected/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-kube-api-access-wlh45\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:28:39.209825 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.209640    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72b6e169-e4d8-4a5d-b5a2-88d4c424503e-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:28:39.685795 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.685762    4294 generic.go:356] "Generic (PLEG): container finished" podID="72b6e169-e4d8-4a5d-b5a2-88d4c424503e" containerID="15c419604dc515d6a1bc875013b3abd7fdfd89642fc82d30be32aa48ca3aaa22" exitCode=0
Nov 02 23:28:39.686025 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.685813    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"
Nov 02 23:28:39.686025 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.685840    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws" event={"ID":"72b6e169-e4d8-4a5d-b5a2-88d4c424503e","Type":"ContainerDied","Data":"15c419604dc515d6a1bc875013b3abd7fdfd89642fc82d30be32aa48ca3aaa22"}
Nov 02 23:28:39.686025 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.685898    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws" event={"ID":"72b6e169-e4d8-4a5d-b5a2-88d4c424503e","Type":"ContainerDied","Data":"e8b82da3f26b258cc5af6b9dca5abf69376d8fd5bc427c5fb12492ed0b505a64"}
Nov 02 23:28:39.686025 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.685910    4294 scope.go:122] "RemoveContainer" containerID="15c419604dc515d6a1bc875013b3abd7fdfd89642fc82d30be32aa48ca3aaa22"
Nov 02 23:28:39.696197 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.696178    4294 scope.go:122] "RemoveContainer" containerID="15c419604dc515d6a1bc875013b3abd7fdfd89642fc82d30be32aa48ca3aaa22"
Nov 02 23:28:39.696479 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:28:39.696451    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c419604dc515d6a1bc875013b3abd7fdfd89642fc82d30be32aa48ca3aaa22\": container with ID starting with 15c419604dc515d6a1bc875013b3abd7fdfd89642fc82d30be32aa48ca3aaa22 not found: ID does not exist" containerID="15c419604dc515d6a1bc875013b3abd7fdfd89642fc82d30be32aa48ca3aaa22"
Nov 02 23:28:39.696533 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.696487    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c419604dc515d6a1bc875013b3abd7fdfd89642fc82d30be32aa48ca3aaa22"} err="failed to get container status \"15c419604dc515d6a1bc875013b3abd7fdfd89642fc82d30be32aa48ca3aaa22\": rpc error: code = NotFound desc = could not find container \"15c419604dc515d6a1bc875013b3abd7fdfd89642fc82d30be32aa48ca3aaa22\": container with ID starting with 15c419604dc515d6a1bc875013b3abd7fdfd89642fc82d30be32aa48ca3aaa22 not found: ID does not exist"
Nov 02 23:28:39.850339 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.844972    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"]
Nov 02 23:28:39.890658 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:39.890619    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-hnvws"]
Nov 02 23:28:40.547878 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:40.547845    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b6e169-e4d8-4a5d-b5a2-88d4c424503e" path="/var/lib/kubelet/pods/72b6e169-e4d8-4a5d-b5a2-88d4c424503e/volumes"
Nov 02 23:28:41.751528 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:28:41.751489    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:28:42.343613 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:42.343578    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:45.997565 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:45.997533    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:47.109737 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.109702    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:47.583486 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.583452    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"]
Nov 02 23:28:47.583964 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.583952    4294 cpu_manager.go:414] "containerMap: removing container" podUID="72b6e169-e4d8-4a5d-b5a2-88d4c424503e" containerName="istio-proxy"
Nov 02 23:28:47.584028 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.583966    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b6e169-e4d8-4a5d-b5a2-88d4c424503e" containerName="istio-proxy"
Nov 02 23:28:47.584028 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.584019    4294 cpu_manager.go:414] "containerMap: removing container" podUID="28a17c69-bf9c-4920-b804-429af4fbfadf" containerName="discovery"
Nov 02 23:28:47.584028 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.584026    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a17c69-bf9c-4920-b804-429af4fbfadf" containerName="discovery"
Nov 02 23:28:47.584127 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.584122    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b6e169-e4d8-4a5d-b5a2-88d4c424503e" containerName="istio-proxy"
Nov 02 23:28:47.584168 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.584134    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a17c69-bf9c-4920-b804-429af4fbfadf" containerName="discovery"
Nov 02 23:28:47.589607 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.589588    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.641766 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.641739    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"]
Nov 02 23:28:47.700692 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.700662    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:47.703344 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.703323    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-02cdd9-openshift-default-dockercfg-5s2hs\""
Nov 02 23:28:47.703657 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.703638    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pq95\" (UniqueName: \"kubernetes.io/projected/7f6e952c-2543-4730-b9f3-598d55c51d0d-kube-api-access-2pq95\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.703722 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.703671    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-podinfo\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.703842 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.703828    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-token\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.703907 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.703848    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-envoy\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.703907 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.703873    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f6e952c-2543-4730-b9f3-598d55c51d0d-istiod-ca-cert\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.704010 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.703953    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-workload-certs\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.704062 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.704028    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-credential-socket\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.704169 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.704154    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-data\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.704215 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.704179    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-workload-socket\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.805372 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.805334    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-data\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.805583 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.805385    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-workload-socket\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.805583 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.805502    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pq95\" (UniqueName: \"kubernetes.io/projected/7f6e952c-2543-4730-b9f3-598d55c51d0d-kube-api-access-2pq95\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.805583 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.805539    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-podinfo\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.805783 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.805589    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-token\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.805783 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.805616    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-envoy\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.805783 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.805652    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f6e952c-2543-4730-b9f3-598d55c51d0d-istiod-ca-cert\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.805783 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.805694    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-workload-certs\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.805783 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.805720    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-credential-socket\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.806073 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.805809    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-data\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.806520 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.806484    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-credential-socket\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.806520 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.806513    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-workload-socket\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.806732 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.806713    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f6e952c-2543-4730-b9f3-598d55c51d0d-istiod-ca-cert\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.806790 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.806768    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-workload-certs\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.808512 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.808483    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-envoy\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.808637 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.808618    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-podinfo\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.860775 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.860711    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-token\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.860897 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.860870    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pq95\" (UniqueName: \"kubernetes.io/projected/7f6e952c-2543-4730-b9f3-598d55c51d0d-kube-api-access-2pq95\") pod \"e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") " pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:47.867079 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.867055    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:47.905274 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:47.905247    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:48.116296 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:48.116214    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:48.321600 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:48.321564    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"]
Nov 02 23:28:48.355847 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:28:48.355822    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6e952c_2543_4730_b9f3_598d55c51d0d.slice/crio-e473a20072cebb386e1cf0698f5d843bcd8e5f1ae32915fea937f160460dbbb1 WatchSource:0}: Error finding container e473a20072cebb386e1cf0698f5d843bcd8e5f1ae32915fea937f160460dbbb1: Status 404 returned error can't find the container with id e473a20072cebb386e1cf0698f5d843bcd8e5f1ae32915fea937f160460dbbb1
Nov 02 23:28:48.358202 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:48.358174    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:28:48.358273 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:48.358239    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:28:48.358273 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:48.358268    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:28:48.740310 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:48.740225    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f" event={"ID":"7f6e952c-2543-4730-b9f3-598d55c51d0d","Type":"ContainerStarted","Data":"4168679baef5fe0ab3ad49f409530fa8296d7f92283cc2ec83440b469a791767"}
Nov 02 23:28:48.740310 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:48.740279    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f" event={"ID":"7f6e952c-2543-4730-b9f3-598d55c51d0d","Type":"ContainerStarted","Data":"e473a20072cebb386e1cf0698f5d843bcd8e5f1ae32915fea937f160460dbbb1"}
Nov 02 23:28:48.906126 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:48.906097    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:48.945662 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:48.945612    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f" podStartSLOduration=1.945599171 podStartE2EDuration="1.945599171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:28:48.897348747 +0000 UTC m=+2168.465703556" watchObservedRunningTime="2026-11-02 23:28:48.945599171 +0000 UTC m=+2168.513953957"
Nov 02 23:28:49.911493 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:49.911460    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:50.756816 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:50.756780    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:50.757857 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:50.757832    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:28:51.649746 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:51.649712    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:51.827785 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:51.827749    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:51.993838 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:51.993803    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:54.836177 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:54.836143    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:55.005040 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.005007    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:55.162110 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.162004    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:28:55.225602 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.225561    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl"]
Nov 02 23:28:55.229822 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.229801    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl"
Nov 02 23:28:55.280800 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.280771    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl"]
Nov 02 23:28:55.285230 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.285202    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c7b45d-5085-430a-aec1-d4d09eb35e52-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl\" (UID: \"c9c7b45d-5085-430a-aec1-d4d09eb35e52\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl"
Nov 02 23:28:55.285397 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.285370    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsg22\" (UniqueName: \"kubernetes.io/projected/c9c7b45d-5085-430a-aec1-d4d09eb35e52-kube-api-access-qsg22\") pod \"payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl\" (UID: \"c9c7b45d-5085-430a-aec1-d4d09eb35e52\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl"
Nov 02 23:28:55.325139 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.325093    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-02cdd9\""
Nov 02 23:28:55.325348 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.325286    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-02cdd9-dockercfg-qmqkr\""
Nov 02 23:28:55.386309 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.386277    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs"]
Nov 02 23:28:55.386798 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.386775    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c7b45d-5085-430a-aec1-d4d09eb35e52-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl\" (UID: \"c9c7b45d-5085-430a-aec1-d4d09eb35e52\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl"
Nov 02 23:28:55.386895 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.386880    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsg22\" (UniqueName: \"kubernetes.io/projected/c9c7b45d-5085-430a-aec1-d4d09eb35e52-kube-api-access-qsg22\") pod \"payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl\" (UID: \"c9c7b45d-5085-430a-aec1-d4d09eb35e52\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl"
Nov 02 23:28:55.387704 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.387683    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c7b45d-5085-430a-aec1-d4d09eb35e52-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl\" (UID: \"c9c7b45d-5085-430a-aec1-d4d09eb35e52\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl"
Nov 02 23:28:55.390623 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.390607    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs"
Nov 02 23:28:55.447959 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.447894    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs"]
Nov 02 23:28:55.451675 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.451649    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsg22\" (UniqueName: \"kubernetes.io/projected/c9c7b45d-5085-430a-aec1-d4d09eb35e52-kube-api-access-qsg22\") pod \"payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl\" (UID: \"c9c7b45d-5085-430a-aec1-d4d09eb35e52\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl"
Nov 02 23:28:55.488477 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.488451    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlttq\" (UniqueName: \"kubernetes.io/projected/6b218d62-c9f7-46d6-b1fc-23a20569ecd7-kube-api-access-jlttq\") pod \"payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs\" (UID: \"6b218d62-c9f7-46d6-b1fc-23a20569ecd7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs"
Nov 02 23:28:55.488609 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.488498    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b218d62-c9f7-46d6-b1fc-23a20569ecd7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs\" (UID: \"6b218d62-c9f7-46d6-b1fc-23a20569ecd7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs"
Nov 02 23:28:55.543768 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.543742    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl"
Nov 02 23:28:55.589721 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.589683    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b218d62-c9f7-46d6-b1fc-23a20569ecd7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs\" (UID: \"6b218d62-c9f7-46d6-b1fc-23a20569ecd7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs"
Nov 02 23:28:55.589934 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.589883    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlttq\" (UniqueName: \"kubernetes.io/projected/6b218d62-c9f7-46d6-b1fc-23a20569ecd7-kube-api-access-jlttq\") pod \"payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs\" (UID: \"6b218d62-c9f7-46d6-b1fc-23a20569ecd7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs"
Nov 02 23:28:55.590677 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.590656    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b218d62-c9f7-46d6-b1fc-23a20569ecd7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs\" (UID: \"6b218d62-c9f7-46d6-b1fc-23a20569ecd7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs"
Nov 02 23:28:55.643946 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.643918    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlttq\" (UniqueName: \"kubernetes.io/projected/6b218d62-c9f7-46d6-b1fc-23a20569ecd7-kube-api-access-jlttq\") pod \"payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs\" (UID: \"6b218d62-c9f7-46d6-b1fc-23a20569ecd7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs"
Nov 02 23:28:55.704530 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.704451    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs"
Nov 02 23:28:55.908631 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:55.908609    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl"]
Nov 02 23:28:55.909448 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:28:55.909422    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c7b45d_5085_430a_aec1_d4d09eb35e52.slice/crio-1cb9b539afdf156f19f69e37fb40e0e8d12a40a738c8272297a6cf1fb713d67d WatchSource:0}: Error finding container 1cb9b539afdf156f19f69e37fb40e0e8d12a40a738c8272297a6cf1fb713d67d: Status 404 returned error can't find the container with id 1cb9b539afdf156f19f69e37fb40e0e8d12a40a738c8272297a6cf1fb713d67d
Nov 02 23:28:56.076831 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:56.076807    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs"]
Nov 02 23:28:56.077044 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:28:56.077020    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b218d62_c9f7_46d6_b1fc_23a20569ecd7.slice/crio-ceb415e8f97cff28f70cfe2290212ac95994ab9c339bedb09efb1f826b9fff84 WatchSource:0}: Error finding container ceb415e8f97cff28f70cfe2290212ac95994ab9c339bedb09efb1f826b9fff84: Status 404 returned error can't find the container with id ceb415e8f97cff28f70cfe2290212ac95994ab9c339bedb09efb1f826b9fff84
Nov 02 23:28:56.804149 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:56.804108    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl" event={"ID":"c9c7b45d-5085-430a-aec1-d4d09eb35e52","Type":"ContainerStarted","Data":"489fb1578c48645e4d33e402c136dd875cadadd1e34a93c1131f36c76c6bcb1e"}
Nov 02 23:28:56.804352 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:56.804204    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl" event={"ID":"c9c7b45d-5085-430a-aec1-d4d09eb35e52","Type":"ContainerStarted","Data":"1cb9b539afdf156f19f69e37fb40e0e8d12a40a738c8272297a6cf1fb713d67d"}
Nov 02 23:28:56.804352 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:56.804235    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl"
Nov 02 23:28:56.805674 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:56.805648    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs" event={"ID":"6b218d62-c9f7-46d6-b1fc-23a20569ecd7","Type":"ContainerStarted","Data":"9ac3b5f6a4580e49da35f6d7ef18a06a5d21135a13c2716427ffc845402046dc"}
Nov 02 23:28:56.805820 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:56.805685    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs" event={"ID":"6b218d62-c9f7-46d6-b1fc-23a20569ecd7","Type":"ContainerStarted","Data":"ceb415e8f97cff28f70cfe2290212ac95994ab9c339bedb09efb1f826b9fff84"}
Nov 02 23:28:56.805820 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:56.805721    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs"
Nov 02 23:28:56.913082 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:56.913036    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl" podStartSLOduration=1.913023452 podStartE2EDuration="1.913023452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:28:56.912546801 +0000 UTC m=+2176.480901588" watchObservedRunningTime="2026-11-02 23:28:56.913023452 +0000 UTC m=+2176.481378239"
Nov 02 23:28:57.023339 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:57.023282    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs" podStartSLOduration=2.023263982 podStartE2EDuration="2.023263982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:28:57.020540974 +0000 UTC m=+2176.588895762" watchObservedRunningTime="2026-11-02 23:28:57.023263982 +0000 UTC m=+2176.591618771"
Nov 02 23:28:59.984333 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:28:59.984303    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:04.897075 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:04.897037    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:07.814315 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:07.814287    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs"
Nov 02 23:29:07.814766 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:07.814349    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl"
Nov 02 23:29:09.476153 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:09.476121    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"]
Nov 02 23:29:09.480166 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:09.480149    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"
Nov 02 23:29:09.532872 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:09.532841    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"]
Nov 02 23:29:09.553855 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:09.553828    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln\" (UID: \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"
Nov 02 23:29:09.554000 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:09.553916    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrnvw\" (UniqueName: \"kubernetes.io/projected/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-kube-api-access-mrnvw\") pod \"maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln\" (UID: \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"
Nov 02 23:29:09.554081 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:09.554008    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-metrics-certs\") pod \"maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln\" (UID: \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"
Nov 02 23:29:09.554310 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:09.554284    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln\" (UID: \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"
Nov 02 23:29:09.576488 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:09.576462    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-1c14e6-serving-cert\""
Nov 02 23:29:09.655346 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:09.655315    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln\" (UID: \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"
Nov 02 23:29:09.655557 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:09.655377    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln\" (UID: \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"
Nov 02 23:29:09.655557 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:09.655411    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrnvw\" (UniqueName: \"kubernetes.io/projected/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-kube-api-access-mrnvw\") pod \"maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln\" (UID: \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"
Nov 02 23:29:09.655557 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:09.655464    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-metrics-certs\") pod \"maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln\" (UID: \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"
Nov 02 23:29:09.655726 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:29:09.655555    4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-serving-cert: secret "maas-api-e2e-praxis-b-1c14e6-serving-cert" not found
Nov 02 23:29:09.655726 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:29:09.655613    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-maas-api-tls podName:23d8c8d7-67c7-4175-9bcd-d21aed75f9d5 nodeName:}" failed. No retries permitted until 2026-11-02 23:29:10.155597967 +0000 UTC m=+2189.723952754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-maas-api-tls") pod "maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln" (UID: "23d8c8d7-67c7-4175-9bcd-d21aed75f9d5") : secret "maas-api-e2e-praxis-b-1c14e6-serving-cert" not found
Nov 02 23:29:09.656027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:09.656004    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln\" (UID: \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"
Nov 02 23:29:09.666091 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:09.666055    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-metrics-certs\") pod \"maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln\" (UID: \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"
Nov 02 23:29:09.717062 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:09.717036    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrnvw\" (UniqueName: \"kubernetes.io/projected/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-kube-api-access-mrnvw\") pod \"maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln\" (UID: \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"
Nov 02 23:29:09.720967 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:09.720928    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:09.884870 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:09.884837    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:10.055380 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:10.055345    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:10.161740 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:10.161668    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln\" (UID: \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"
Nov 02 23:29:10.163867 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:10.163839    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln\" (UID: \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"
Nov 02 23:29:10.393778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:10.393749    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"
Nov 02 23:29:10.763513 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:29:10.763484    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d8c8d7_67c7_4175_9bcd_d21aed75f9d5.slice/crio-74ebfaf25351eded17675ed0a667a8faf1f0beed217f135086d295206c344009 WatchSource:0}: Error finding container 74ebfaf25351eded17675ed0a667a8faf1f0beed217f135086d295206c344009: Status 404 returned error can't find the container with id 74ebfaf25351eded17675ed0a667a8faf1f0beed217f135086d295206c344009
Nov 02 23:29:10.763912 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:10.763659    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"]
Nov 02 23:29:10.896821 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:10.896790    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln" event={"ID":"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5","Type":"ContainerStarted","Data":"74ebfaf25351eded17675ed0a667a8faf1f0beed217f135086d295206c344009"}
Nov 02 23:29:11.754373 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:29:11.754343    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:29:11.903735 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:11.903701    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln" event={"ID":"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5","Type":"ContainerStarted","Data":"518ef8fa04c609371d2e65ec96f37315de52d6c30356c37222920c012518522a"}
Nov 02 23:29:11.904139 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:11.903783    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"
Nov 02 23:29:12.020310 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:12.020257    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln" podStartSLOduration=2.107392256 podStartE2EDuration="3.020241219s" totalImagesPullingTime="912.848963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:29:12.018304317 +0000 UTC m=+2191.586659102" watchObservedRunningTime="2026-11-02 23:29:12.020241219 +0000 UTC m=+2191.588596006"
Nov 02 23:29:14.827164 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:14.827127    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:17.914877 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:17.914850    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"
Nov 02 23:29:18.163178 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:18.163145    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:19.731227 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:19.731191    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:21.212139 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:21.212103    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:21.712208 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:21.712171    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:21.886697 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:21.886665    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:25.199137 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:25.199063    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"]
Nov 02 23:29:25.199621 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:25.199270    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b" podUID="4238eef4-75f3-4b97-b84e-dcd47d4931c0" containerName="istio-proxy" containerID="cri-o://467675910c95bbcafc4b723a4411c2aecbd10e4c5f55dc390052df3743194133" gracePeriod=30
Nov 02 23:29:26.272241 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:26.272201    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:28.334738 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:28.334691    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:28.507452 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:28.507400    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:29.952621 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:29.952588    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:30.442824 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.442794    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:30.483262 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.483190    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:29:30.600928 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.600895    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-workload-certs\") pod \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") "
Nov 02 23:29:30.601160 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.600955    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istiod-ca-cert\") pod \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") "
Nov 02 23:29:30.601160 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.601025    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-workload-socket\") pod \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") "
Nov 02 23:29:30.601160 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.601051    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-credential-socket\") pod \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") "
Nov 02 23:29:30.601326 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.601227    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-workload-certs" pod "4238eef4-75f3-4b97-b84e-dcd47d4931c0" (UID: "4238eef4-75f3-4b97-b84e-dcd47d4931c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:29:30.601326 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.601271    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-data\") pod \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") "
Nov 02 23:29:30.601326 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.601280    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-workload-socket" pod "4238eef4-75f3-4b97-b84e-dcd47d4931c0" (UID: "4238eef4-75f3-4b97-b84e-dcd47d4931c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:29:30.601326 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.601305    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-token\" (UniqueName: \"kubernetes.io/projected/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-token\") pod \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") "
Nov 02 23:29:30.601522 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.601306    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-credential-socket" pod "4238eef4-75f3-4b97-b84e-dcd47d4931c0" (UID: "4238eef4-75f3-4b97-b84e-dcd47d4931c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:29:30.601522 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.601343    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-envoy\") pod \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") "
Nov 02 23:29:30.601522 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.601399    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4238eef4-75f3-4b97-b84e-dcd47d4931c0-kube-api-access-wcl92\" (UniqueName: \"kubernetes.io/projected/4238eef4-75f3-4b97-b84e-dcd47d4931c0-kube-api-access-wcl92\") pod \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") "
Nov 02 23:29:30.601522 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.601404    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istiod-ca-cert" pod "4238eef4-75f3-4b97-b84e-dcd47d4931c0" (UID: "4238eef4-75f3-4b97-b84e-dcd47d4931c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:29:30.601522 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.601449    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-data" pod "4238eef4-75f3-4b97-b84e-dcd47d4931c0" (UID: "4238eef4-75f3-4b97-b84e-dcd47d4931c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:29:30.601522 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.601440    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-podinfo\") pod \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\" (UID: \"4238eef4-75f3-4b97-b84e-dcd47d4931c0\") "
Nov 02 23:29:30.602093 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.602001    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:29:30.602093 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.602025    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:29:30.602093 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.602042    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:29:30.602093 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.602057    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:29:30.602093 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.602069    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:29:30.603594 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.603570    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-envoy" pod "4238eef4-75f3-4b97-b84e-dcd47d4931c0" (UID: "4238eef4-75f3-4b97-b84e-dcd47d4931c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:29:30.603702 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.603599    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-token" pod "4238eef4-75f3-4b97-b84e-dcd47d4931c0" (UID: "4238eef4-75f3-4b97-b84e-dcd47d4931c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:29:30.603872 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.603853    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-podinfo" pod "4238eef4-75f3-4b97-b84e-dcd47d4931c0" (UID: "4238eef4-75f3-4b97-b84e-dcd47d4931c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:29:30.603945 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.603931    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4238eef4-75f3-4b97-b84e-dcd47d4931c0-kube-api-access-wcl92" pod "4238eef4-75f3-4b97-b84e-dcd47d4931c0" (UID: "4238eef4-75f3-4b97-b84e-dcd47d4931c0"). InnerVolumeSpecName "kube-api-access-wcl92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:29:30.703228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.703194    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:29:30.703228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.703222    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:29:30.703453 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.703234    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcl92\" (UniqueName: \"kubernetes.io/projected/4238eef4-75f3-4b97-b84e-dcd47d4931c0-kube-api-access-wcl92\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:29:30.703453 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:30.703247    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4238eef4-75f3-4b97-b84e-dcd47d4931c0-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:29:31.007316 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:31.007280    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:31.013120 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:31.013096    4294 generic.go:356] "Generic (PLEG): container finished" podID="4238eef4-75f3-4b97-b84e-dcd47d4931c0" containerID="467675910c95bbcafc4b723a4411c2aecbd10e4c5f55dc390052df3743194133" exitCode=0
Nov 02 23:29:31.013258 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:31.013145    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"
Nov 02 23:29:31.013258 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:31.013174    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b" event={"ID":"4238eef4-75f3-4b97-b84e-dcd47d4931c0","Type":"ContainerDied","Data":"467675910c95bbcafc4b723a4411c2aecbd10e4c5f55dc390052df3743194133"}
Nov 02 23:29:31.013258 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:31.013220    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b" event={"ID":"4238eef4-75f3-4b97-b84e-dcd47d4931c0","Type":"ContainerDied","Data":"8b50975328345d0b461a097890cbeb04c9953383036a808ab09155239f0f930c"}
Nov 02 23:29:31.013258 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:31.013243    4294 scope.go:122] "RemoveContainer" containerID="467675910c95bbcafc4b723a4411c2aecbd10e4c5f55dc390052df3743194133"
Nov 02 23:29:31.028200 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:31.028175    4294 scope.go:122] "RemoveContainer" containerID="467675910c95bbcafc4b723a4411c2aecbd10e4c5f55dc390052df3743194133"
Nov 02 23:29:31.028461 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:29:31.028441    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"467675910c95bbcafc4b723a4411c2aecbd10e4c5f55dc390052df3743194133\": container with ID starting with 467675910c95bbcafc4b723a4411c2aecbd10e4c5f55dc390052df3743194133 not found: ID does not exist" containerID="467675910c95bbcafc4b723a4411c2aecbd10e4c5f55dc390052df3743194133"
Nov 02 23:29:31.028530 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:31.028468    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467675910c95bbcafc4b723a4411c2aecbd10e4c5f55dc390052df3743194133"} err="failed to get container status \"467675910c95bbcafc4b723a4411c2aecbd10e4c5f55dc390052df3743194133\": rpc error: code = NotFound desc = could not find container \"467675910c95bbcafc4b723a4411c2aecbd10e4c5f55dc390052df3743194133\": container with ID starting with 467675910c95bbcafc4b723a4411c2aecbd10e4c5f55dc390052df3743194133 not found: ID does not exist"
Nov 02 23:29:31.172815 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:31.172782    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"]
Nov 02 23:29:31.178179 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:31.178155    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-v4j5b"]
Nov 02 23:29:31.375622 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:31.375545    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:31.564322 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:31.564291    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:31.735164 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:31.735091    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:32.122549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:32.122482    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:32.338733 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:32.338689    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:32.548093 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:32.548061    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4238eef4-75f3-4b97-b84e-dcd47d4931c0" path="/var/lib/kubelet/pods/4238eef4-75f3-4b97-b84e-dcd47d4931c0/volumes"
Nov 02 23:29:33.050288 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:33.050250    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:33.891749 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:33.891675    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:34.784288 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:34.784246    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:35.069773 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:35.069669    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:36.287659 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:36.287631    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:36.772116 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:36.772078    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:38.000897 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:38.000867    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:38.235266 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:38.235234    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:38.417436 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:38.417323    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:38.606821 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:38.606774    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:38.812272 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:38.812232    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:39.111531 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:39.111446    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:41.216822 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:41.216791    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:41.371092 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:41.371010    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:41.751505 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:29:41.751472    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:29:42.854582 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:42.854552    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:43.055636 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:43.055600    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:43.917576 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:43.917542    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:46.009460 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:46.009428    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:46.174902 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:46.174869    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:49.484485 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:49.484453    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:55.961888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:55.961854    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:56.127957 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:56.127926    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:29:56.281966 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:29:56.281932    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:00.187570 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.187536    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp"]
Nov 02 23:30:00.188183 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.188167    4294 cpu_manager.go:414] "containerMap: removing container" podUID="4238eef4-75f3-4b97-b84e-dcd47d4931c0" containerName="istio-proxy"
Nov 02 23:30:00.188242 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.188185    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4238eef4-75f3-4b97-b84e-dcd47d4931c0" containerName="istio-proxy"
Nov 02 23:30:00.188282 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.188272    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4238eef4-75f3-4b97-b84e-dcd47d4931c0" containerName="istio-proxy"
Nov 02 23:30:00.193298 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.193275    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp"
Nov 02 23:30:00.206321 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.206296    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt"]
Nov 02 23:30:00.210126 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.210109    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt"
Nov 02 23:30:00.262723 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.262683    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwh5j\" (UniqueName: \"kubernetes.io/projected/94a70e85-c3ec-45b1-b025-558b6aaaa187-kube-api-access-vwh5j\") pod \"maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp\" (UID: \"94a70e85-c3ec-45b1-b025-558b6aaaa187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp"
Nov 02 23:30:00.262885 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.262788    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a70e85-c3ec-45b1-b025-558b6aaaa187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp\" (UID: \"94a70e85-c3ec-45b1-b025-558b6aaaa187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp"
Nov 02 23:30:00.308548 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.306514    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp"]
Nov 02 23:30:00.312417 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.312386    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt"]
Nov 02 23:30:00.349238 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.349202    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\""
Nov 02 23:30:00.363928 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.363897    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a70e85-c3ec-45b1-b025-558b6aaaa187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp\" (UID: \"94a70e85-c3ec-45b1-b025-558b6aaaa187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp"
Nov 02 23:30:00.364089 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.364007    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67d6383-e114-48a5-a32d-ef31d124705f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt\" (UID: \"c67d6383-e114-48a5-a32d-ef31d124705f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt"
Nov 02 23:30:00.364138 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.364087    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2d47\" (UniqueName: \"kubernetes.io/projected/c67d6383-e114-48a5-a32d-ef31d124705f-kube-api-access-h2d47\") pod \"maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt\" (UID: \"c67d6383-e114-48a5-a32d-ef31d124705f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt"
Nov 02 23:30:00.364251 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.364236    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwh5j\" (UniqueName: \"kubernetes.io/projected/94a70e85-c3ec-45b1-b025-558b6aaaa187-kube-api-access-vwh5j\") pod \"maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp\" (UID: \"94a70e85-c3ec-45b1-b025-558b6aaaa187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp"
Nov 02 23:30:00.364529 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.364509    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a70e85-c3ec-45b1-b025-558b6aaaa187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp\" (UID: \"94a70e85-c3ec-45b1-b025-558b6aaaa187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp"
Nov 02 23:30:00.419542 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.419513    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwh5j\" (UniqueName: \"kubernetes.io/projected/94a70e85-c3ec-45b1-b025-558b6aaaa187-kube-api-access-vwh5j\") pod \"maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp\" (UID: \"94a70e85-c3ec-45b1-b025-558b6aaaa187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp"
Nov 02 23:30:00.465772 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.465697    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67d6383-e114-48a5-a32d-ef31d124705f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt\" (UID: \"c67d6383-e114-48a5-a32d-ef31d124705f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt"
Nov 02 23:30:00.465772 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.465746    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2d47\" (UniqueName: \"kubernetes.io/projected/c67d6383-e114-48a5-a32d-ef31d124705f-kube-api-access-h2d47\") pod \"maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt\" (UID: \"c67d6383-e114-48a5-a32d-ef31d124705f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt"
Nov 02 23:30:00.466355 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.466336    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67d6383-e114-48a5-a32d-ef31d124705f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt\" (UID: \"c67d6383-e114-48a5-a32d-ef31d124705f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt"
Nov 02 23:30:00.507574 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.507540    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp"
Nov 02 23:30:00.523817 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.520411    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2d47\" (UniqueName: \"kubernetes.io/projected/c67d6383-e114-48a5-a32d-ef31d124705f-kube-api-access-h2d47\") pod \"maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt\" (UID: \"c67d6383-e114-48a5-a32d-ef31d124705f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt"
Nov 02 23:30:00.523817 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.523767    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt"
Nov 02 23:30:00.883424 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.883396    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp"]
Nov 02 23:30:00.883634 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:30:00.883600    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a70e85_c3ec_45b1_b025_558b6aaaa187.slice/crio-86b5f304c6fabef23d5430f7eee33ad0d76cd2b5fa6ac82e06595dea2be4d476 WatchSource:0}: Error finding container 86b5f304c6fabef23d5430f7eee33ad0d76cd2b5fa6ac82e06595dea2be4d476: Status 404 returned error can't find the container with id 86b5f304c6fabef23d5430f7eee33ad0d76cd2b5fa6ac82e06595dea2be4d476
Nov 02 23:30:00.928521 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:00.928492    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt"]
Nov 02 23:30:00.928951 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:30:00.928923    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67d6383_e114_48a5_a32d_ef31d124705f.slice/crio-006c0f549892f52c05d5662ace43e2cc98f860713a7b3c82e4f2995897990a5d WatchSource:0}: Error finding container 006c0f549892f52c05d5662ace43e2cc98f860713a7b3c82e4f2995897990a5d: Status 404 returned error can't find the container with id 006c0f549892f52c05d5662ace43e2cc98f860713a7b3c82e4f2995897990a5d
Nov 02 23:30:01.088080 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:01.088048    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:01.200237 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:01.200202    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt" event={"ID":"c67d6383-e114-48a5-a32d-ef31d124705f","Type":"ContainerStarted","Data":"006c0f549892f52c05d5662ace43e2cc98f860713a7b3c82e4f2995897990a5d"}
Nov 02 23:30:01.201704 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:01.201682    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp" event={"ID":"94a70e85-c3ec-45b1-b025-558b6aaaa187","Type":"ContainerStarted","Data":"eb2b078bbdb8b3622817e04d0929a707ff06fd8ae7f9b8cea1528c5df14ae72c"}
Nov 02 23:30:01.201820 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:01.201724    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp" event={"ID":"94a70e85-c3ec-45b1-b025-558b6aaaa187","Type":"ContainerStarted","Data":"86b5f304c6fabef23d5430f7eee33ad0d76cd2b5fa6ac82e06595dea2be4d476"}
Nov 02 23:30:01.314255 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:01.314199    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp" podStartSLOduration=1.314187185 podStartE2EDuration="1.314187185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:01.31184929 +0000 UTC m=+2240.880204078" watchObservedRunningTime="2026-11-02 23:30:01.314187185 +0000 UTC m=+2240.882541972"
Nov 02 23:30:02.209070 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:02.208963    4294 generic.go:356] "Generic (PLEG): container finished" podID="c67d6383-e114-48a5-a32d-ef31d124705f" containerID="1f3b0490964d4c1fc9565dbed178c0ada15e9fc26d79a7ff7486927e8f30954b" exitCode=60
Nov 02 23:30:02.209070 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:02.209027    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt" event={"ID":"c67d6383-e114-48a5-a32d-ef31d124705f","Type":"ContainerDied","Data":"1f3b0490964d4c1fc9565dbed178c0ada15e9fc26d79a7ff7486927e8f30954b"}
Nov 02 23:30:02.209551 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:02.209421    4294 scope.go:122] "RemoveContainer" containerID="1f3b0490964d4c1fc9565dbed178c0ada15e9fc26d79a7ff7486927e8f30954b"
Nov 02 23:30:02.210818 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:02.210793    4294 generic.go:356] "Generic (PLEG): container finished" podID="94a70e85-c3ec-45b1-b025-558b6aaaa187" containerID="eb2b078bbdb8b3622817e04d0929a707ff06fd8ae7f9b8cea1528c5df14ae72c" exitCode=60
Nov 02 23:30:02.210927 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:02.210867    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp" event={"ID":"94a70e85-c3ec-45b1-b025-558b6aaaa187","Type":"ContainerDied","Data":"eb2b078bbdb8b3622817e04d0929a707ff06fd8ae7f9b8cea1528c5df14ae72c"}
Nov 02 23:30:02.211113 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:02.211096    4294 scope.go:122] "RemoveContainer" containerID="eb2b078bbdb8b3622817e04d0929a707ff06fd8ae7f9b8cea1528c5df14ae72c"
Nov 02 23:30:03.219729 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:03.219688    4294 generic.go:356] "Generic (PLEG): container finished" podID="c67d6383-e114-48a5-a32d-ef31d124705f" containerID="ac652fbf03bcf19bdeaeaeecee98fcb9bc5dc6de479356b88de1648ad3099c47" exitCode=60
Nov 02 23:30:03.220238 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:03.219773    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt" event={"ID":"c67d6383-e114-48a5-a32d-ef31d124705f","Type":"ContainerDied","Data":"ac652fbf03bcf19bdeaeaeecee98fcb9bc5dc6de479356b88de1648ad3099c47"}
Nov 02 23:30:03.220238 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:03.219854    4294 scope.go:122] "RemoveContainer" containerID="1f3b0490964d4c1fc9565dbed178c0ada15e9fc26d79a7ff7486927e8f30954b"
Nov 02 23:30:03.220238 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:03.220177    4294 scope.go:122] "RemoveContainer" containerID="ac652fbf03bcf19bdeaeaeecee98fcb9bc5dc6de479356b88de1648ad3099c47"
Nov 02 23:30:03.220480 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:30:03.220462    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt_odh-ai-gateway-infra(c67d6383-e114-48a5-a32d-ef31d124705f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt" podUID="c67d6383-e114-48a5-a32d-ef31d124705f"
Nov 02 23:30:03.222542 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:03.222510    4294 generic.go:356] "Generic (PLEG): container finished" podID="94a70e85-c3ec-45b1-b025-558b6aaaa187" containerID="0599302b199aa88fa24ac52d003949905a9cbf3703603bddaa945cbf10a7120e" exitCode=60
Nov 02 23:30:03.222676 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:03.222570    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp" event={"ID":"94a70e85-c3ec-45b1-b025-558b6aaaa187","Type":"ContainerDied","Data":"0599302b199aa88fa24ac52d003949905a9cbf3703603bddaa945cbf10a7120e"}
Nov 02 23:30:03.222841 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:03.222828    4294 scope.go:122] "RemoveContainer" containerID="0599302b199aa88fa24ac52d003949905a9cbf3703603bddaa945cbf10a7120e"
Nov 02 23:30:03.223120 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:30:03.223103    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp_odh-ai-gateway-infra(94a70e85-c3ec-45b1-b025-558b6aaaa187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp" podUID="94a70e85-c3ec-45b1-b025-558b6aaaa187"
Nov 02 23:30:03.240127 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:03.240093    4294 scope.go:122] "RemoveContainer" containerID="eb2b078bbdb8b3622817e04d0929a707ff06fd8ae7f9b8cea1528c5df14ae72c"
Nov 02 23:30:04.230761 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:04.230731    4294 scope.go:122] "RemoveContainer" containerID="0599302b199aa88fa24ac52d003949905a9cbf3703603bddaa945cbf10a7120e"
Nov 02 23:30:04.231253 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:30:04.231069    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp_odh-ai-gateway-infra(94a70e85-c3ec-45b1-b025-558b6aaaa187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp" podUID="94a70e85-c3ec-45b1-b025-558b6aaaa187"
Nov 02 23:30:04.232480 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:04.232463    4294 scope.go:122] "RemoveContainer" containerID="ac652fbf03bcf19bdeaeaeecee98fcb9bc5dc6de479356b88de1648ad3099c47"
Nov 02 23:30:04.232695 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:30:04.232675    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt_odh-ai-gateway-infra(c67d6383-e114-48a5-a32d-ef31d124705f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt" podUID="c67d6383-e114-48a5-a32d-ef31d124705f"
Nov 02 23:30:06.052164 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:06.052134    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:10.893610 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:10.893557    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:11.755261 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:30:11.755224    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:30:14.540541 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:14.540509    4294 scope.go:122] "RemoveContainer" containerID="0599302b199aa88fa24ac52d003949905a9cbf3703603bddaa945cbf10a7120e"
Nov 02 23:30:14.540949 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:14.540837    4294 scope.go:122] "RemoveContainer" containerID="ac652fbf03bcf19bdeaeaeecee98fcb9bc5dc6de479356b88de1648ad3099c47"
Nov 02 23:30:15.305356 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:15.305319    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp" event={"ID":"94a70e85-c3ec-45b1-b025-558b6aaaa187","Type":"ContainerStarted","Data":"f81963ed3beac5135224902835531cb6c7d04f9c9b70ce698b2f62499a2964fd"}
Nov 02 23:30:15.307401 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:15.307369    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt" event={"ID":"c67d6383-e114-48a5-a32d-ef31d124705f","Type":"ContainerStarted","Data":"181e944eaa1b7894cf4d5fbcb4135cfe7815cce5551989288d110a15049eb528"}
Nov 02 23:30:15.538768 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:15.538713    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt" podStartSLOduration=15.538698598 podStartE2EDuration="15.538698598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:15.536329199 +0000 UTC m=+2255.104683986" watchObservedRunningTime="2026-11-02 23:30:15.538698598 +0000 UTC m=+2255.107053385"
Nov 02 23:30:15.712469 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:15.712438    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:16.314890 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:16.314857    4294 generic.go:356] "Generic (PLEG): container finished" podID="94a70e85-c3ec-45b1-b025-558b6aaaa187" containerID="f81963ed3beac5135224902835531cb6c7d04f9c9b70ce698b2f62499a2964fd" exitCode=60
Nov 02 23:30:16.315117 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:16.314925    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp" event={"ID":"94a70e85-c3ec-45b1-b025-558b6aaaa187","Type":"ContainerDied","Data":"f81963ed3beac5135224902835531cb6c7d04f9c9b70ce698b2f62499a2964fd"}
Nov 02 23:30:16.315117 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:16.315001    4294 scope.go:122] "RemoveContainer" containerID="0599302b199aa88fa24ac52d003949905a9cbf3703603bddaa945cbf10a7120e"
Nov 02 23:30:16.315406 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:16.315380    4294 scope.go:122] "RemoveContainer" containerID="f81963ed3beac5135224902835531cb6c7d04f9c9b70ce698b2f62499a2964fd"
Nov 02 23:30:16.315720 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:30:16.315697    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp_odh-ai-gateway-infra(94a70e85-c3ec-45b1-b025-558b6aaaa187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp" podUID="94a70e85-c3ec-45b1-b025-558b6aaaa187"
Nov 02 23:30:16.317155 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:16.317136    4294 generic.go:356] "Generic (PLEG): container finished" podID="c67d6383-e114-48a5-a32d-ef31d124705f" containerID="181e944eaa1b7894cf4d5fbcb4135cfe7815cce5551989288d110a15049eb528" exitCode=60
Nov 02 23:30:16.317240 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:16.317169    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt" event={"ID":"c67d6383-e114-48a5-a32d-ef31d124705f","Type":"ContainerDied","Data":"181e944eaa1b7894cf4d5fbcb4135cfe7815cce5551989288d110a15049eb528"}
Nov 02 23:30:16.317497 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:16.317484    4294 scope.go:122] "RemoveContainer" containerID="181e944eaa1b7894cf4d5fbcb4135cfe7815cce5551989288d110a15049eb528"
Nov 02 23:30:16.317736 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:30:16.317720    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt_odh-ai-gateway-infra(c67d6383-e114-48a5-a32d-ef31d124705f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt" podUID="c67d6383-e114-48a5-a32d-ef31d124705f"
Nov 02 23:30:16.327278 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:16.327258    4294 scope.go:122] "RemoveContainer" containerID="ac652fbf03bcf19bdeaeaeecee98fcb9bc5dc6de479356b88de1648ad3099c47"
Nov 02 23:30:16.443245 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:16.443215    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp"]
Nov 02 23:30:16.582316 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:16.582217    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt"]
Nov 02 23:30:17.551865 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:17.551841    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp"
Nov 02 23:30:17.554071 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:17.554051    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt"
Nov 02 23:30:17.595835 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:17.595773    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a70e85-c3ec-45b1-b025-558b6aaaa187-kube-api-access-vwh5j\" (UniqueName: \"kubernetes.io/projected/94a70e85-c3ec-45b1-b025-558b6aaaa187-kube-api-access-vwh5j\") pod \"94a70e85-c3ec-45b1-b025-558b6aaaa187\" (UID: \"94a70e85-c3ec-45b1-b025-558b6aaaa187\") "
Nov 02 23:30:17.596003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:17.595856    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a70e85-c3ec-45b1-b025-558b6aaaa187-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a70e85-c3ec-45b1-b025-558b6aaaa187-maas-api-service-ca\") pod \"94a70e85-c3ec-45b1-b025-558b6aaaa187\" (UID: \"94a70e85-c3ec-45b1-b025-558b6aaaa187\") "
Nov 02 23:30:17.596003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:17.595918    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c67d6383-e114-48a5-a32d-ef31d124705f-kube-api-access-h2d47\" (UniqueName: \"kubernetes.io/projected/c67d6383-e114-48a5-a32d-ef31d124705f-kube-api-access-h2d47\") pod \"c67d6383-e114-48a5-a32d-ef31d124705f\" (UID: \"c67d6383-e114-48a5-a32d-ef31d124705f\") "
Nov 02 23:30:17.596139 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:17.596052    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c67d6383-e114-48a5-a32d-ef31d124705f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67d6383-e114-48a5-a32d-ef31d124705f-maas-api-service-ca\") pod \"c67d6383-e114-48a5-a32d-ef31d124705f\" (UID: \"c67d6383-e114-48a5-a32d-ef31d124705f\") "
Nov 02 23:30:17.596343 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:17.596253    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a70e85-c3ec-45b1-b025-558b6aaaa187-maas-api-service-ca" pod "94a70e85-c3ec-45b1-b025-558b6aaaa187" (UID: "94a70e85-c3ec-45b1-b025-558b6aaaa187"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:30:17.596464 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:17.596401    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c67d6383-e114-48a5-a32d-ef31d124705f-maas-api-service-ca" pod "c67d6383-e114-48a5-a32d-ef31d124705f" (UID: "c67d6383-e114-48a5-a32d-ef31d124705f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:30:17.596549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:17.596535    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67d6383-e114-48a5-a32d-ef31d124705f-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:30:17.596602 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:17.596553    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94a70e85-c3ec-45b1-b025-558b6aaaa187-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:30:17.597834 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:17.597813    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a70e85-c3ec-45b1-b025-558b6aaaa187-kube-api-access-vwh5j" pod "94a70e85-c3ec-45b1-b025-558b6aaaa187" (UID: "94a70e85-c3ec-45b1-b025-558b6aaaa187"). InnerVolumeSpecName "kube-api-access-vwh5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:30:17.597922 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:17.597904    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67d6383-e114-48a5-a32d-ef31d124705f-kube-api-access-h2d47" pod "c67d6383-e114-48a5-a32d-ef31d124705f" (UID: "c67d6383-e114-48a5-a32d-ef31d124705f"). InnerVolumeSpecName "kube-api-access-h2d47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:30:17.697963 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:17.697920    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2d47\" (UniqueName: \"kubernetes.io/projected/c67d6383-e114-48a5-a32d-ef31d124705f-kube-api-access-h2d47\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:30:17.697963 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:17.697953    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwh5j\" (UniqueName: \"kubernetes.io/projected/94a70e85-c3ec-45b1-b025-558b6aaaa187-kube-api-access-vwh5j\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:30:18.333854 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:18.333826    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt"
Nov 02 23:30:18.334062 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:18.333867    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt" event={"ID":"c67d6383-e114-48a5-a32d-ef31d124705f","Type":"ContainerDied","Data":"006c0f549892f52c05d5662ace43e2cc98f860713a7b3c82e4f2995897990a5d"}
Nov 02 23:30:18.334062 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:18.333941    4294 scope.go:122] "RemoveContainer" containerID="181e944eaa1b7894cf4d5fbcb4135cfe7815cce5551989288d110a15049eb528"
Nov 02 23:30:18.335458 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:18.335435    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp" event={"ID":"94a70e85-c3ec-45b1-b025-558b6aaaa187","Type":"ContainerDied","Data":"86b5f304c6fabef23d5430f7eee33ad0d76cd2b5fa6ac82e06595dea2be4d476"}
Nov 02 23:30:18.335625 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:18.335448    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp"
Nov 02 23:30:18.345060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:18.345043    4294 scope.go:122] "RemoveContainer" containerID="f81963ed3beac5135224902835531cb6c7d04f9c9b70ce698b2f62499a2964fd"
Nov 02 23:30:18.504792 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:18.504760    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp"]
Nov 02 23:30:18.507691 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:18.507662    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-81c3c9-29894370-fnftp"]
Nov 02 23:30:18.549288 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:18.549258    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a70e85-c3ec-45b1-b025-558b6aaaa187" path="/var/lib/kubelet/pods/94a70e85-c3ec-45b1-b025-558b6aaaa187/volumes"
Nov 02 23:30:18.614626 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:18.614547    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt"]
Nov 02 23:30:18.620285 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:18.620255    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-3bf62f-29894370-jltbt"]
Nov 02 23:30:20.549498 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:20.549459    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c67d6383-e114-48a5-a32d-ef31d124705f" path="/var/lib/kubelet/pods/c67d6383-e114-48a5-a32d-ef31d124705f/volumes"
Nov 02 23:30:20.881940 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:20.881911    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:20.981422 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:20.981388    4294 patch_prober.go:29] interesting pod/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": read tcp 10.133.0.2:48284->10.133.0.112:15021: read: connection reset by peer" start-of-body=
Nov 02 23:30:20.981577 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:20.981460    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f" podUID="7f6e952c-2543-4730-b9f3-598d55c51d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": read tcp 10.133.0.2:48284->10.133.0.112:15021: read: connection reset by peer"
Nov 02 23:30:21.359330 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:21.359297    4294 generic.go:356] "Generic (PLEG): container finished" podID="7f6e952c-2543-4730-b9f3-598d55c51d0d" containerID="4168679baef5fe0ab3ad49f409530fa8296d7f92283cc2ec83440b469a791767" exitCode=0
Nov 02 23:30:21.359501 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:21.359369    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f" event={"ID":"7f6e952c-2543-4730-b9f3-598d55c51d0d","Type":"ContainerDied","Data":"4168679baef5fe0ab3ad49f409530fa8296d7f92283cc2ec83440b469a791767"}
Nov 02 23:30:21.359894 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:21.359880    4294 scope.go:122] "RemoveContainer" containerID="4168679baef5fe0ab3ad49f409530fa8296d7f92283cc2ec83440b469a791767"
Nov 02 23:30:21.361268 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:21.361243    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:30:21.361333 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:21.361308    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:30:21.361368 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:21.361339    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:30:22.375002 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:22.371338    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f" event={"ID":"7f6e952c-2543-4730-b9f3-598d55c51d0d","Type":"ContainerStarted","Data":"0a927e3faaf5ec7932b28558559309c72676f755fa042299e386e8b0db03203c"}
Nov 02 23:30:22.905912 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:22.905885    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:30:22.906140 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:22.906024    4294 patch_prober.go:29] interesting pod/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 02 23:30:22.906140 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:22.906078    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f" podUID="7f6e952c-2543-4730-b9f3-598d55c51d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 02 23:30:23.906558 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:23.906518    4294 patch_prober.go:29] interesting pod/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 02 23:30:23.906963 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:23.906582    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f" podUID="7f6e952c-2543-4730-b9f3-598d55c51d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 02 23:30:24.749512 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:24.749480    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:24.905914 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:24.905879    4294 patch_prober.go:29] interesting pod/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 02 23:30:24.906126 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:24.905933    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f" podUID="7f6e952c-2543-4730-b9f3-598d55c51d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 02 23:30:25.281907 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:25.281878    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:25.906500 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:25.906467    4294 patch_prober.go:29] interesting pod/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 02 23:30:25.906683 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:25.906515    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f" podUID="7f6e952c-2543-4730-b9f3-598d55c51d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 02 23:30:25.979236 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:25.979197    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:26.906253 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:26.906215    4294 patch_prober.go:29] interesting pod/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 02 23:30:26.906666 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:26.906275    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f" podUID="7f6e952c-2543-4730-b9f3-598d55c51d0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 02 23:30:27.909594 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:27.909563    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:30:27.910111 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:27.909918    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:30:27.910532 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:27.910511    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:30:28.378181 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:28.378149    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:30.379215 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:30.379182    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:32.035379 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.035340    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"]
Nov 02 23:30:32.035909 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.035895    4294 cpu_manager.go:414] "containerMap: removing container" podUID="c67d6383-e114-48a5-a32d-ef31d124705f" containerName="cleanup"
Nov 02 23:30:32.035909 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.035911    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c67d6383-e114-48a5-a32d-ef31d124705f" containerName="cleanup"
Nov 02 23:30:32.036031 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.035923    4294 cpu_manager.go:414] "containerMap: removing container" podUID="94a70e85-c3ec-45b1-b025-558b6aaaa187" containerName="cleanup"
Nov 02 23:30:32.036031 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.035928    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a70e85-c3ec-45b1-b025-558b6aaaa187" containerName="cleanup"
Nov 02 23:30:32.036031 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.035942    4294 cpu_manager.go:414] "containerMap: removing container" podUID="c67d6383-e114-48a5-a32d-ef31d124705f" containerName="cleanup"
Nov 02 23:30:32.036031 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.035946    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c67d6383-e114-48a5-a32d-ef31d124705f" containerName="cleanup"
Nov 02 23:30:32.036031 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.035967    4294 cpu_manager.go:414] "containerMap: removing container" podUID="94a70e85-c3ec-45b1-b025-558b6aaaa187" containerName="cleanup"
Nov 02 23:30:32.036031 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.035972    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a70e85-c3ec-45b1-b025-558b6aaaa187" containerName="cleanup"
Nov 02 23:30:32.036031 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.035999    4294 cpu_manager.go:414] "containerMap: removing container" podUID="c67d6383-e114-48a5-a32d-ef31d124705f" containerName="cleanup"
Nov 02 23:30:32.036031 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.036003    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c67d6383-e114-48a5-a32d-ef31d124705f" containerName="cleanup"
Nov 02 23:30:32.036031 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.036011    4294 cpu_manager.go:414] "containerMap: removing container" podUID="94a70e85-c3ec-45b1-b025-558b6aaaa187" containerName="cleanup"
Nov 02 23:30:32.036031 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.036015    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a70e85-c3ec-45b1-b025-558b6aaaa187" containerName="cleanup"
Nov 02 23:30:32.036361 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.036088    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c67d6383-e114-48a5-a32d-ef31d124705f" containerName="cleanup"
Nov 02 23:30:32.036361 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.036096    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a70e85-c3ec-45b1-b025-558b6aaaa187" containerName="cleanup"
Nov 02 23:30:32.036361 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.036104    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c67d6383-e114-48a5-a32d-ef31d124705f" containerName="cleanup"
Nov 02 23:30:32.036361 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.036116    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a70e85-c3ec-45b1-b025-558b6aaaa187" containerName="cleanup"
Nov 02 23:30:32.038407 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.038392    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:32.136304 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.136268    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-81c3c9\"/\"default-dockercfg-5c7cd\""
Nov 02 23:30:32.136472 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.136381    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-81c3c9\"/\"kube-root-ca.crt\""
Nov 02 23:30:32.136472 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.136404    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-81c3c9\"/\"openshift-service-ca.crt\""
Nov 02 23:30:32.136561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.136517    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-81c3c9\"/\"praxis-route-81c3c9-kserve-self-signed-certs\""
Nov 02 23:30:32.139364 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.139346    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"]
Nov 02 23:30:32.149550 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.149528    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:32.194702 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.194656    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-dshm\") pod \"praxis-route-81c3c9-kserve-58bd655467-fp9kh\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") " pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:32.194894 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.194716    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-model-cache\") pod \"praxis-route-81c3c9-kserve-58bd655467-fp9kh\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") " pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:32.194894 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.194837    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0141f50-637d-4e3f-af89-02424a5aa910-tls-certs\") pod \"praxis-route-81c3c9-kserve-58bd655467-fp9kh\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") " pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:32.195028 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.195004    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-tmp-dir\") pod \"praxis-route-81c3c9-kserve-58bd655467-fp9kh\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") " pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:32.195138 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.195121    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzmbx\" (UniqueName: \"kubernetes.io/projected/e0141f50-637d-4e3f-af89-02424a5aa910-kube-api-access-zzmbx\") pod \"praxis-route-81c3c9-kserve-58bd655467-fp9kh\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") " pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:32.195193 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.195179    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-home\") pod \"praxis-route-81c3c9-kserve-58bd655467-fp9kh\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") " pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:32.296881 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.296805    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-tmp-dir\") pod \"praxis-route-81c3c9-kserve-58bd655467-fp9kh\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") " pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:32.296881 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.296864    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzmbx\" (UniqueName: \"kubernetes.io/projected/e0141f50-637d-4e3f-af89-02424a5aa910-kube-api-access-zzmbx\") pod \"praxis-route-81c3c9-kserve-58bd655467-fp9kh\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") " pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:32.297162 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.296894    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-home\") pod \"praxis-route-81c3c9-kserve-58bd655467-fp9kh\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") " pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:32.297162 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.296947    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-dshm\") pod \"praxis-route-81c3c9-kserve-58bd655467-fp9kh\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") " pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:32.297162 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.296970    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-model-cache\") pod \"praxis-route-81c3c9-kserve-58bd655467-fp9kh\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") " pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:32.297162 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.297027    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0141f50-637d-4e3f-af89-02424a5aa910-tls-certs\") pod \"praxis-route-81c3c9-kserve-58bd655467-fp9kh\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") " pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:32.297367 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.297301    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-tmp-dir\") pod \"praxis-route-81c3c9-kserve-58bd655467-fp9kh\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") " pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:32.297367 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.297332    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-home\") pod \"praxis-route-81c3c9-kserve-58bd655467-fp9kh\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") " pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:32.297439 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.297390    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-model-cache\") pod \"praxis-route-81c3c9-kserve-58bd655467-fp9kh\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") " pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:32.299161 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.299136    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-dshm\") pod \"praxis-route-81c3c9-kserve-58bd655467-fp9kh\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") " pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:32.299423 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.299403    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0141f50-637d-4e3f-af89-02424a5aa910-tls-certs\") pod \"praxis-route-81c3c9-kserve-58bd655467-fp9kh\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") " pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:32.353117 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.353088    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzmbx\" (UniqueName: \"kubernetes.io/projected/e0141f50-637d-4e3f-af89-02424a5aa910-kube-api-access-zzmbx\") pod \"praxis-route-81c3c9-kserve-58bd655467-fp9kh\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") " pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:32.649402 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:32.649321    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:33.014510 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:33.014472    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"]
Nov 02 23:30:33.014622 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:30:33.014557    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0141f50_637d_4e3f_af89_02424a5aa910.slice/crio-3e4212681892f4e091656c852b37b9665f259bd85f63feb69e55337498f0c229 WatchSource:0}: Error finding container 3e4212681892f4e091656c852b37b9665f259bd85f63feb69e55337498f0c229: Status 404 returned error can't find the container with id 3e4212681892f4e091656c852b37b9665f259bd85f63feb69e55337498f0c229
Nov 02 23:30:33.460452 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:33.460344    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh" event={"ID":"e0141f50-637d-4e3f-af89-02424a5aa910","Type":"ContainerStarted","Data":"d6b90580849dd61b26172c4d520628f5c5051f32f2cd3ec2eddc1732c9d01449"}
Nov 02 23:30:33.460452 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:33.460423    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh" event={"ID":"e0141f50-637d-4e3f-af89-02424a5aa910","Type":"ContainerStarted","Data":"3e4212681892f4e091656c852b37b9665f259bd85f63feb69e55337498f0c229"}
Nov 02 23:30:33.472880 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:33.472850    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:33.572507 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:33.572455    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh" podStartSLOduration=2.572440883 podStartE2EDuration="2.572440883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:30:33.571119398 +0000 UTC m=+2273.139474187" watchObservedRunningTime="2026-11-02 23:30:33.572440883 +0000 UTC m=+2273.140795670"
Nov 02 23:30:33.649762 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:33.649731    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:34.202598 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:34.202564    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:34.361529 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:34.361499    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:38.375216 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:38.375182    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:41.750888 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:30:41.750846    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:30:42.649778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:42.649743    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:42.662255 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:42.662229    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:43.096701 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:43.096660    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:43.268026 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:43.267993    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:43.423928 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:43.423847    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:43.531631 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:43.531603    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:30:46.232235 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:46.232200    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:48.145189 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:48.145157    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:52.954695 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:52.954655    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:53.119653 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:53.119623    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:53.281066 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:53.281026    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:30:58.016261 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:30:58.016227    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:03.109330 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:03.109296    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:06.659281 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:06.659244    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:10.001804 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:10.001772    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:11.749431 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:31:11.749386    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:31:14.999869 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:14.999836    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:19.792853 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:19.792816    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:20.522346 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:20.522312    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:22.611384 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:22.611344    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:25.605917 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:25.605885    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:28.657384 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:28.657351    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:29.908838 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:29.908802    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc"]
Nov 02 23:31:30.286941 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:30.286901    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:30.451100 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:30.451071    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:30.623928 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:30.623844    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:31.937818 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:31.937781    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"]
Nov 02 23:31:31.938686 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:31.938662    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a70e85-c3ec-45b1-b025-558b6aaaa187" containerName="cleanup"
Nov 02 23:31:31.938802 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:31.938694    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c67d6383-e114-48a5-a32d-ef31d124705f" containerName="cleanup"
Nov 02 23:31:31.942331 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:31.942310    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:31.996820 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:31.996788    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"]
Nov 02 23:31:32.040763 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.040737    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-981724\"/\"ipp-route-981724-kserve-self-signed-certs\""
Nov 02 23:31:32.040763 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.040765    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-981724\"/\"default-dockercfg-bl4s5\""
Nov 02 23:31:32.041705 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.041688    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-981724\"/\"kube-root-ca.crt\""
Nov 02 23:31:32.041765 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.041716    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-981724\"/\"openshift-service-ca.crt\""
Nov 02 23:31:32.061748 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.061709    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:32.097791 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.097754    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-model-cache\") pod \"ipp-route-981724-kserve-7dd9464db5-f85hx\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") " pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:32.098079 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.098057    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-home\") pod \"ipp-route-981724-kserve-7dd9464db5-f85hx\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") " pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:32.098290 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.098267    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c2acb51-b739-4375-a332-13837baa1c01-tls-certs\") pod \"ipp-route-981724-kserve-7dd9464db5-f85hx\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") " pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:32.098424 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.098329    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-tmp-dir\") pod \"ipp-route-981724-kserve-7dd9464db5-f85hx\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") " pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:32.098424 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.098389    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpfdf\" (UniqueName: \"kubernetes.io/projected/2c2acb51-b739-4375-a332-13837baa1c01-kube-api-access-rpfdf\") pod \"ipp-route-981724-kserve-7dd9464db5-f85hx\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") " pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:32.098545 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.098489    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-dshm\") pod \"ipp-route-981724-kserve-7dd9464db5-f85hx\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") " pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:32.199444 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.199348    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-model-cache\") pod \"ipp-route-981724-kserve-7dd9464db5-f85hx\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") " pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:32.199444 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.199393    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-home\") pod \"ipp-route-981724-kserve-7dd9464db5-f85hx\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") " pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:32.199444 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.199433    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c2acb51-b739-4375-a332-13837baa1c01-tls-certs\") pod \"ipp-route-981724-kserve-7dd9464db5-f85hx\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") " pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:32.199734 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.199472    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-tmp-dir\") pod \"ipp-route-981724-kserve-7dd9464db5-f85hx\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") " pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:32.199734 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.199530    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpfdf\" (UniqueName: \"kubernetes.io/projected/2c2acb51-b739-4375-a332-13837baa1c01-kube-api-access-rpfdf\") pod \"ipp-route-981724-kserve-7dd9464db5-f85hx\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") " pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:32.199734 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.199612    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-dshm\") pod \"ipp-route-981724-kserve-7dd9464db5-f85hx\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") " pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:32.199921 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.199788    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-model-cache\") pod \"ipp-route-981724-kserve-7dd9464db5-f85hx\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") " pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:32.199921 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.199868    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-home\") pod \"ipp-route-981724-kserve-7dd9464db5-f85hx\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") " pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:32.199921 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.199878    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-tmp-dir\") pod \"ipp-route-981724-kserve-7dd9464db5-f85hx\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") " pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:32.201758 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.201737    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-dshm\") pod \"ipp-route-981724-kserve-7dd9464db5-f85hx\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") " pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:32.201938 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.201922    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c2acb51-b739-4375-a332-13837baa1c01-tls-certs\") pod \"ipp-route-981724-kserve-7dd9464db5-f85hx\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") " pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:32.254408 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.254384    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpfdf\" (UniqueName: \"kubernetes.io/projected/2c2acb51-b739-4375-a332-13837baa1c01-kube-api-access-rpfdf\") pod \"ipp-route-981724-kserve-7dd9464db5-f85hx\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") " pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:32.554358 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.554331    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:32.926842 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:32.926805    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"]
Nov 02 23:31:32.927252 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:31:32.927225    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c2acb51_b739_4375_a332_13837baa1c01.slice/crio-24ae6dc1267e7fe9b736a9c04085524d785c6948de7ad180d67f3330cc28af64 WatchSource:0}: Error finding container 24ae6dc1267e7fe9b736a9c04085524d785c6948de7ad180d67f3330cc28af64: Status 404 returned error can't find the container with id 24ae6dc1267e7fe9b736a9c04085524d785c6948de7ad180d67f3330cc28af64
Nov 02 23:31:33.836245 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:33.836209    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx" event={"ID":"2c2acb51-b739-4375-a332-13837baa1c01","Type":"ContainerStarted","Data":"ccc3653907efe3dae9b62d319e3b7ca0c8a8053f331be9e794999b2e99cedf62"}
Nov 02 23:31:33.836653 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:33.836259    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx" event={"ID":"2c2acb51-b739-4375-a332-13837baa1c01","Type":"ContainerStarted","Data":"24ae6dc1267e7fe9b736a9c04085524d785c6948de7ad180d67f3330cc28af64"}
Nov 02 23:31:33.948801 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:33.948745    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx" podStartSLOduration=2.948731594 podStartE2EDuration="2.948731594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:31:33.945892492 +0000 UTC m=+2333.514247279" watchObservedRunningTime="2026-11-02 23:31:33.948731594 +0000 UTC m=+2333.517086381"
Nov 02 23:31:34.505376 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:34.505341    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:34.557447 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:34.557415    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:34.671186 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:34.671149    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:34.909343 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:34.909246    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc" podUID="0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d" containerName="payload-processing" containerID="cri-o://784ad7f9a4f08aee1cea128411f7b776dddfcf89b053caf969fcfb9bc929d701" gracePeriod=25
Nov 02 23:31:35.175879 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:35.175814    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc"
Nov 02 23:31:35.233465 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:35.233438    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d-plugins-config-volume\") pod \"0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d\" (UID: \"0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d\") "
Nov 02 23:31:35.233609 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:35.233471    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d-kube-api-access-6m7tb\" (UniqueName: \"kubernetes.io/projected/0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d-kube-api-access-6m7tb\") pod \"0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d\" (UID: \"0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d\") "
Nov 02 23:31:35.233853 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:35.233822    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d-plugins-config-volume" pod "0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d" (UID: "0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:31:35.235543 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:35.235517    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d-kube-api-access-6m7tb" pod "0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d" (UID: "0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d"). InnerVolumeSpecName "kube-api-access-6m7tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:31:35.334882 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:35.334847    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:31:35.334882 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:35.334875    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m7tb\" (UniqueName: \"kubernetes.io/projected/0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d-kube-api-access-6m7tb\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:31:35.848992 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:35.848947    4294 generic.go:356] "Generic (PLEG): container finished" podID="0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d" containerID="784ad7f9a4f08aee1cea128411f7b776dddfcf89b053caf969fcfb9bc929d701" exitCode=0
Nov 02 23:31:35.849201 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:35.849018    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc"
Nov 02 23:31:35.849201 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:35.849023    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc" event={"ID":"0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d","Type":"ContainerDied","Data":"784ad7f9a4f08aee1cea128411f7b776dddfcf89b053caf969fcfb9bc929d701"}
Nov 02 23:31:35.849201 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:35.849076    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc" event={"ID":"0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d","Type":"ContainerDied","Data":"4cff90053b48d676c47a29de246533d6c564447d07656d35f5df6be0fb2eb6cb"}
Nov 02 23:31:35.849201 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:35.849097    4294 scope.go:122] "RemoveContainer" containerID="784ad7f9a4f08aee1cea128411f7b776dddfcf89b053caf969fcfb9bc929d701"
Nov 02 23:31:35.858724 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:35.858706    4294 scope.go:122] "RemoveContainer" containerID="784ad7f9a4f08aee1cea128411f7b776dddfcf89b053caf969fcfb9bc929d701"
Nov 02 23:31:35.859003 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:31:35.858965    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"784ad7f9a4f08aee1cea128411f7b776dddfcf89b053caf969fcfb9bc929d701\": container with ID starting with 784ad7f9a4f08aee1cea128411f7b776dddfcf89b053caf969fcfb9bc929d701 not found: ID does not exist" containerID="784ad7f9a4f08aee1cea128411f7b776dddfcf89b053caf969fcfb9bc929d701"
Nov 02 23:31:35.859059 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:35.859004    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784ad7f9a4f08aee1cea128411f7b776dddfcf89b053caf969fcfb9bc929d701"} err="failed to get container status \"784ad7f9a4f08aee1cea128411f7b776dddfcf89b053caf969fcfb9bc929d701\": rpc error: code = NotFound desc = could not find container \"784ad7f9a4f08aee1cea128411f7b776dddfcf89b053caf969fcfb9bc929d701\": container with ID starting with 784ad7f9a4f08aee1cea128411f7b776dddfcf89b053caf969fcfb9bc929d701 not found: ID does not exist"
Nov 02 23:31:35.874837 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:35.874807    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:36.006328 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:36.006292    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc"]
Nov 02 23:31:36.053922 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:36.053894    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3bf62f-75bd4c8f7c-hx9rc"]
Nov 02 23:31:36.547260 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:36.547228    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d" path="/var/lib/kubelet/pods/0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d/volumes"
Nov 02 23:31:41.750182 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:31:41.750140    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:31:42.459070 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:42.459037    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:42.555312 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:42.555280    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:42.568594 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:42.568564    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:42.905277 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:42.905242    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:31:47.427626 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:47.427591    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:51.040843 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:51.040809    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:51.623186 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:51.623148    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:51.806476 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:51.806436    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:52.402335 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:52.402298    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:31:57.282783 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:31:57.282743    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:01.397966 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:01.397926    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"]
Nov 02 23:32:01.398795 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:01.398237    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4" podUID="44063d97-ef69-410b-af13-2c7896689fc3" containerName="maas-api" containerID="cri-o://97f063b9b22acd81a4b31fdd2cdec1bb43dc68ac1827d3dfc797a43eb92dcd1c" gracePeriod=30
Nov 02 23:32:01.521866 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:01.521836    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:01.678375 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:01.678350    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"
Nov 02 23:32:01.756956 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:01.756922    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44063d97-ef69-410b-af13-2c7896689fc3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44063d97-ef69-410b-af13-2c7896689fc3-maas-api-tls\") pod \"44063d97-ef69-410b-af13-2c7896689fc3\" (UID: \"44063d97-ef69-410b-af13-2c7896689fc3\") "
Nov 02 23:32:01.757162 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:01.757061    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44063d97-ef69-410b-af13-2c7896689fc3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44063d97-ef69-410b-af13-2c7896689fc3-metrics-certs\") pod \"44063d97-ef69-410b-af13-2c7896689fc3\" (UID: \"44063d97-ef69-410b-af13-2c7896689fc3\") "
Nov 02 23:32:01.757162 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:01.757126    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44063d97-ef69-410b-af13-2c7896689fc3-kube-api-access-gr6bw\" (UniqueName: \"kubernetes.io/projected/44063d97-ef69-410b-af13-2c7896689fc3-kube-api-access-gr6bw\") pod \"44063d97-ef69-410b-af13-2c7896689fc3\" (UID: \"44063d97-ef69-410b-af13-2c7896689fc3\") "
Nov 02 23:32:01.757305 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:01.757221    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44063d97-ef69-410b-af13-2c7896689fc3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44063d97-ef69-410b-af13-2c7896689fc3-openshift-service-ca\") pod \"44063d97-ef69-410b-af13-2c7896689fc3\" (UID: \"44063d97-ef69-410b-af13-2c7896689fc3\") "
Nov 02 23:32:01.757596 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:01.757568    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44063d97-ef69-410b-af13-2c7896689fc3-openshift-service-ca" pod "44063d97-ef69-410b-af13-2c7896689fc3" (UID: "44063d97-ef69-410b-af13-2c7896689fc3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:32:01.757735 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:01.757692    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44063d97-ef69-410b-af13-2c7896689fc3-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:32:01.759311 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:01.759286    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44063d97-ef69-410b-af13-2c7896689fc3-metrics-certs" pod "44063d97-ef69-410b-af13-2c7896689fc3" (UID: "44063d97-ef69-410b-af13-2c7896689fc3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:32:01.759420 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:01.759313    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44063d97-ef69-410b-af13-2c7896689fc3-maas-api-tls" pod "44063d97-ef69-410b-af13-2c7896689fc3" (UID: "44063d97-ef69-410b-af13-2c7896689fc3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:32:01.759420 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:01.759360    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44063d97-ef69-410b-af13-2c7896689fc3-kube-api-access-gr6bw" pod "44063d97-ef69-410b-af13-2c7896689fc3" (UID: "44063d97-ef69-410b-af13-2c7896689fc3"). InnerVolumeSpecName "kube-api-access-gr6bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:32:01.859026 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:01.858963    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44063d97-ef69-410b-af13-2c7896689fc3-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:32:01.859026 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:01.859023    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr6bw\" (UniqueName: \"kubernetes.io/projected/44063d97-ef69-410b-af13-2c7896689fc3-kube-api-access-gr6bw\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:32:01.859026 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:01.859034    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44063d97-ef69-410b-af13-2c7896689fc3-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:32:02.004218 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:02.004183    4294 generic.go:356] "Generic (PLEG): container finished" podID="44063d97-ef69-410b-af13-2c7896689fc3" containerID="97f063b9b22acd81a4b31fdd2cdec1bb43dc68ac1827d3dfc797a43eb92dcd1c" exitCode=0
Nov 02 23:32:02.004423 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:02.004227    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"
Nov 02 23:32:02.004423 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:02.004267    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4" event={"ID":"44063d97-ef69-410b-af13-2c7896689fc3","Type":"ContainerDied","Data":"97f063b9b22acd81a4b31fdd2cdec1bb43dc68ac1827d3dfc797a43eb92dcd1c"}
Nov 02 23:32:02.004423 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:02.004325    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4" event={"ID":"44063d97-ef69-410b-af13-2c7896689fc3","Type":"ContainerDied","Data":"932ba42938bf4c6c1cc6a7877b304136a69262ab44b300952d1eb6e6f13cf522"}
Nov 02 23:32:02.004423 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:02.004354    4294 scope.go:122] "RemoveContainer" containerID="97f063b9b22acd81a4b31fdd2cdec1bb43dc68ac1827d3dfc797a43eb92dcd1c"
Nov 02 23:32:02.014795 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:02.014780    4294 scope.go:122] "RemoveContainer" containerID="97f063b9b22acd81a4b31fdd2cdec1bb43dc68ac1827d3dfc797a43eb92dcd1c"
Nov 02 23:32:02.015085 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:32:02.015066    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97f063b9b22acd81a4b31fdd2cdec1bb43dc68ac1827d3dfc797a43eb92dcd1c\": container with ID starting with 97f063b9b22acd81a4b31fdd2cdec1bb43dc68ac1827d3dfc797a43eb92dcd1c not found: ID does not exist" containerID="97f063b9b22acd81a4b31fdd2cdec1bb43dc68ac1827d3dfc797a43eb92dcd1c"
Nov 02 23:32:02.015164 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:02.015096    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97f063b9b22acd81a4b31fdd2cdec1bb43dc68ac1827d3dfc797a43eb92dcd1c"} err="failed to get container status \"97f063b9b22acd81a4b31fdd2cdec1bb43dc68ac1827d3dfc797a43eb92dcd1c\": rpc error: code = NotFound desc = could not find container \"97f063b9b22acd81a4b31fdd2cdec1bb43dc68ac1827d3dfc797a43eb92dcd1c\": container with ID starting with 97f063b9b22acd81a4b31fdd2cdec1bb43dc68ac1827d3dfc797a43eb92dcd1c not found: ID does not exist"
Nov 02 23:32:02.161312 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:02.161276    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"]
Nov 02 23:32:02.210966 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:02.210926    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3bf62f-5df457858b-96km4"]
Nov 02 23:32:02.548459 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:02.548420    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44063d97-ef69-410b-af13-2c7896689fc3" path="/var/lib/kubelet/pods/44063d97-ef69-410b-af13-2c7896689fc3/volumes"
Nov 02 23:32:03.420965 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:03.420931    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:05.065631 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:05.065600    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:05.924596 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:05.924565    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:06.288608 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:06.288568    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:06.470305 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:06.470271    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:06.653673 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:06.653579    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:06.931109 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:06.931030    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:07.977246 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:07.977212    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:08.577614 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:08.577579    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:08.857271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:08.857195    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:09.335393 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:09.335349    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:10.568324 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:10.568290    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:11.489387 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:11.489354    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:11.672528 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:11.672493    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:11.752632 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:32:11.752599    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:32:11.867170 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:11.867142    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:13.050180 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:13.050145    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:13.216425 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:13.216386    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:14.353587 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:14.353543    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:14.645969 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:14.645890    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:14.807528 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:14.807486    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:15.029751 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.029722    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r"]
Nov 02 23:32:15.030272 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.030258    4294 cpu_manager.go:414] "containerMap: removing container" podUID="44063d97-ef69-410b-af13-2c7896689fc3" containerName="maas-api"
Nov 02 23:32:15.030325 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.030273    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44063d97-ef69-410b-af13-2c7896689fc3" containerName="maas-api"
Nov 02 23:32:15.030325 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.030300    4294 cpu_manager.go:414] "containerMap: removing container" podUID="0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d" containerName="payload-processing"
Nov 02 23:32:15.030325 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.030306    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d" containerName="payload-processing"
Nov 02 23:32:15.030424 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.030381    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="44063d97-ef69-410b-af13-2c7896689fc3" containerName="maas-api"
Nov 02 23:32:15.030424 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.030395    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb1a960-15cf-492e-90a4-eb2cd0fb1b6d" containerName="payload-processing"
Nov 02 23:32:15.033438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.033421    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r"
Nov 02 23:32:15.086901 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.086868    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r"]
Nov 02 23:32:15.118166 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.118130    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbedb5c3-3260-41bb-ba5e-50d6e518b458-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r\" (UID: \"cbedb5c3-3260-41bb-ba5e-50d6e518b458\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r"
Nov 02 23:32:15.118357 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.118221    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8q7s\" (UniqueName: \"kubernetes.io/projected/cbedb5c3-3260-41bb-ba5e-50d6e518b458-kube-api-access-q8q7s\") pod \"payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r\" (UID: \"cbedb5c3-3260-41bb-ba5e-50d6e518b458\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r"
Nov 02 23:32:15.131236 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.131205    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-db5d87\""
Nov 02 23:32:15.131417 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.131313    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-db5d87-dockercfg-9qdtq\""
Nov 02 23:32:15.192731 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.192693    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h"]
Nov 02 23:32:15.196439 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.196418    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h"
Nov 02 23:32:15.219514 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.219489    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8q7s\" (UniqueName: \"kubernetes.io/projected/cbedb5c3-3260-41bb-ba5e-50d6e518b458-kube-api-access-q8q7s\") pod \"payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r\" (UID: \"cbedb5c3-3260-41bb-ba5e-50d6e518b458\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r"
Nov 02 23:32:15.219660 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.219641    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbedb5c3-3260-41bb-ba5e-50d6e518b458-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r\" (UID: \"cbedb5c3-3260-41bb-ba5e-50d6e518b458\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r"
Nov 02 23:32:15.220296 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.220279    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbedb5c3-3260-41bb-ba5e-50d6e518b458-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r\" (UID: \"cbedb5c3-3260-41bb-ba5e-50d6e518b458\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r"
Nov 02 23:32:15.249770 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.249742    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h"]
Nov 02 23:32:15.273493 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.273470    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8q7s\" (UniqueName: \"kubernetes.io/projected/cbedb5c3-3260-41bb-ba5e-50d6e518b458-kube-api-access-q8q7s\") pod \"payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r\" (UID: \"cbedb5c3-3260-41bb-ba5e-50d6e518b458\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r"
Nov 02 23:32:15.321356 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.321289    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8402d643-1fb6-4d39-862f-f702b0f685a2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h\" (UID: \"8402d643-1fb6-4d39-862f-f702b0f685a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h"
Nov 02 23:32:15.321356 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.321340    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqq58\" (UniqueName: \"kubernetes.io/projected/8402d643-1fb6-4d39-862f-f702b0f685a2-kube-api-access-pqq58\") pod \"payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h\" (UID: \"8402d643-1fb6-4d39-862f-f702b0f685a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h"
Nov 02 23:32:15.346866 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.346841    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r"
Nov 02 23:32:15.423924 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.423888    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8402d643-1fb6-4d39-862f-f702b0f685a2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h\" (UID: \"8402d643-1fb6-4d39-862f-f702b0f685a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h"
Nov 02 23:32:15.424366 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.423970    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqq58\" (UniqueName: \"kubernetes.io/projected/8402d643-1fb6-4d39-862f-f702b0f685a2-kube-api-access-pqq58\") pod \"payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h\" (UID: \"8402d643-1fb6-4d39-862f-f702b0f685a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h"
Nov 02 23:32:15.425320 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.425291    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8402d643-1fb6-4d39-862f-f702b0f685a2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h\" (UID: \"8402d643-1fb6-4d39-862f-f702b0f685a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h"
Nov 02 23:32:15.478611 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.478584    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqq58\" (UniqueName: \"kubernetes.io/projected/8402d643-1fb6-4d39-862f-f702b0f685a2-kube-api-access-pqq58\") pod \"payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h\" (UID: \"8402d643-1fb6-4d39-862f-f702b0f685a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h"
Nov 02 23:32:15.509165 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.509138    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h"
Nov 02 23:32:15.678627 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.678559    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs"]
Nov 02 23:32:15.698986 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.698958    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:15.723106 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.723084    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r"]
Nov 02 23:32:15.723233 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:32:15.723212    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbedb5c3_3260_41bb_ba5e_50d6e518b458.slice/crio-99a683287cf50aa713ff299610203b2d0bcbc7d2748f50a20efbc3b9d31c8786 WatchSource:0}: Error finding container 99a683287cf50aa713ff299610203b2d0bcbc7d2748f50a20efbc3b9d31c8786: Status 404 returned error can't find the container with id 99a683287cf50aa713ff299610203b2d0bcbc7d2748f50a20efbc3b9d31c8786
Nov 02 23:32:15.780478 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.780451    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl"]
Nov 02 23:32:15.877304 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:15.877277    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h"]
Nov 02 23:32:15.878262 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:32:15.878233    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8402d643_1fb6_4d39_862f_f702b0f685a2.slice/crio-3334f8f2f27b378e7a903ec279fea0f46f17f406937d800303e605a47d59ac12 WatchSource:0}: Error finding container 3334f8f2f27b378e7a903ec279fea0f46f17f406937d800303e605a47d59ac12: Status 404 returned error can't find the container with id 3334f8f2f27b378e7a903ec279fea0f46f17f406937d800303e605a47d59ac12
Nov 02 23:32:16.053880 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:16.053848    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:16.100637 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:16.100598    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r" event={"ID":"cbedb5c3-3260-41bb-ba5e-50d6e518b458","Type":"ContainerStarted","Data":"4f0ee58e7e7d853f43ed1c13432dcde18f46ca33cb0b75340e72624f3dac9d68"}
Nov 02 23:32:16.100852 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:16.100702    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r" event={"ID":"cbedb5c3-3260-41bb-ba5e-50d6e518b458","Type":"ContainerStarted","Data":"99a683287cf50aa713ff299610203b2d0bcbc7d2748f50a20efbc3b9d31c8786"}
Nov 02 23:32:16.100973 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:16.100949    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r"
Nov 02 23:32:16.102941 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:16.102902    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h" event={"ID":"8402d643-1fb6-4d39-862f-f702b0f685a2","Type":"ContainerStarted","Data":"13a947fc667e45b6c54498593c52b5d9e51bec381f61fcf0079ba62634e26dd6"}
Nov 02 23:32:16.103091 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:16.102964    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h"
Nov 02 23:32:16.103091 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:16.103025    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h" event={"ID":"8402d643-1fb6-4d39-862f-f702b0f685a2","Type":"ContainerStarted","Data":"3334f8f2f27b378e7a903ec279fea0f46f17f406937d800303e605a47d59ac12"}
Nov 02 23:32:16.209914 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:16.209814    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r" podStartSLOduration=2.209800503 podStartE2EDuration="2.209800503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:16.208796626 +0000 UTC m=+2375.777151415" watchObservedRunningTime="2026-11-02 23:32:16.209800503 +0000 UTC m=+2375.778155329"
Nov 02 23:32:16.216371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:16.216341    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:16.318862 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:16.318810    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h" podStartSLOduration=1.318796949 podStartE2EDuration="1.318796949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:16.316703214 +0000 UTC m=+2375.885057983" watchObservedRunningTime="2026-11-02 23:32:16.318796949 +0000 UTC m=+2375.887151735"
Nov 02 23:32:16.924997 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:16.924959    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:17.112597 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:17.112562    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:17.274861 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:17.274824    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:20.679130 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:20.679086    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs" podUID="6b218d62-c9f7-46d6-b1fc-23a20569ecd7" containerName="payload-processing" containerID="cri-o://9ac3b5f6a4580e49da35f6d7ef18a06a5d21135a13c2716427ffc845402046dc" gracePeriod=25
Nov 02 23:32:20.781384 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:20.781321    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl" podUID="c9c7b45d-5085-430a-aec1-d4d09eb35e52" containerName="payload-pre-processing" containerID="cri-o://489fb1578c48645e4d33e402c136dd875cadadd1e34a93c1131f36c76c6bcb1e" gracePeriod=25
Nov 02 23:32:20.959517 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:20.959488    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs"
Nov 02 23:32:21.005464 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.005416    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b218d62-c9f7-46d6-b1fc-23a20569ecd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b218d62-c9f7-46d6-b1fc-23a20569ecd7-plugins-config-volume\") pod \"6b218d62-c9f7-46d6-b1fc-23a20569ecd7\" (UID: \"6b218d62-c9f7-46d6-b1fc-23a20569ecd7\") "
Nov 02 23:32:21.005643 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.005509    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b218d62-c9f7-46d6-b1fc-23a20569ecd7-kube-api-access-jlttq\" (UniqueName: \"kubernetes.io/projected/6b218d62-c9f7-46d6-b1fc-23a20569ecd7-kube-api-access-jlttq\") pod \"6b218d62-c9f7-46d6-b1fc-23a20569ecd7\" (UID: \"6b218d62-c9f7-46d6-b1fc-23a20569ecd7\") "
Nov 02 23:32:21.005880 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.005848    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b218d62-c9f7-46d6-b1fc-23a20569ecd7-plugins-config-volume" pod "6b218d62-c9f7-46d6-b1fc-23a20569ecd7" (UID: "6b218d62-c9f7-46d6-b1fc-23a20569ecd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:32:21.006215 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.006193    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b218d62-c9f7-46d6-b1fc-23a20569ecd7-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:32:21.007617 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.007587    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b218d62-c9f7-46d6-b1fc-23a20569ecd7-kube-api-access-jlttq" pod "6b218d62-c9f7-46d6-b1fc-23a20569ecd7" (UID: "6b218d62-c9f7-46d6-b1fc-23a20569ecd7"). InnerVolumeSpecName "kube-api-access-jlttq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:32:21.017113 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.016857    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:21.049311 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.049292    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl"
Nov 02 23:32:21.106965 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.106928    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c7b45d-5085-430a-aec1-d4d09eb35e52-kube-api-access-qsg22\" (UniqueName: \"kubernetes.io/projected/c9c7b45d-5085-430a-aec1-d4d09eb35e52-kube-api-access-qsg22\") pod \"c9c7b45d-5085-430a-aec1-d4d09eb35e52\" (UID: \"c9c7b45d-5085-430a-aec1-d4d09eb35e52\") "
Nov 02 23:32:21.108132 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.107088    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c7b45d-5085-430a-aec1-d4d09eb35e52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c7b45d-5085-430a-aec1-d4d09eb35e52-plugins-config-volume\") pod \"c9c7b45d-5085-430a-aec1-d4d09eb35e52\" (UID: \"c9c7b45d-5085-430a-aec1-d4d09eb35e52\") "
Nov 02 23:32:21.108132 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.107528    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlttq\" (UniqueName: \"kubernetes.io/projected/6b218d62-c9f7-46d6-b1fc-23a20569ecd7-kube-api-access-jlttq\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:32:21.108132 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.107856    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c7b45d-5085-430a-aec1-d4d09eb35e52-plugins-config-volume" pod "c9c7b45d-5085-430a-aec1-d4d09eb35e52" (UID: "c9c7b45d-5085-430a-aec1-d4d09eb35e52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:32:21.109811 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.109789    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c7b45d-5085-430a-aec1-d4d09eb35e52-kube-api-access-qsg22" pod "c9c7b45d-5085-430a-aec1-d4d09eb35e52" (UID: "c9c7b45d-5085-430a-aec1-d4d09eb35e52"). InnerVolumeSpecName "kube-api-access-qsg22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:32:21.134704 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.134677    4294 generic.go:356] "Generic (PLEG): container finished" podID="c9c7b45d-5085-430a-aec1-d4d09eb35e52" containerID="489fb1578c48645e4d33e402c136dd875cadadd1e34a93c1131f36c76c6bcb1e" exitCode=0
Nov 02 23:32:21.134830 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.134723    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl"
Nov 02 23:32:21.134830 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.134767    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl" event={"ID":"c9c7b45d-5085-430a-aec1-d4d09eb35e52","Type":"ContainerDied","Data":"489fb1578c48645e4d33e402c136dd875cadadd1e34a93c1131f36c76c6bcb1e"}
Nov 02 23:32:21.134923 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.134855    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl" event={"ID":"c9c7b45d-5085-430a-aec1-d4d09eb35e52","Type":"ContainerDied","Data":"1cb9b539afdf156f19f69e37fb40e0e8d12a40a738c8272297a6cf1fb713d67d"}
Nov 02 23:32:21.134923 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.134894    4294 scope.go:122] "RemoveContainer" containerID="489fb1578c48645e4d33e402c136dd875cadadd1e34a93c1131f36c76c6bcb1e"
Nov 02 23:32:21.136157 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.136135    4294 generic.go:356] "Generic (PLEG): container finished" podID="6b218d62-c9f7-46d6-b1fc-23a20569ecd7" containerID="9ac3b5f6a4580e49da35f6d7ef18a06a5d21135a13c2716427ffc845402046dc" exitCode=0
Nov 02 23:32:21.136266 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.136165    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs" event={"ID":"6b218d62-c9f7-46d6-b1fc-23a20569ecd7","Type":"ContainerDied","Data":"9ac3b5f6a4580e49da35f6d7ef18a06a5d21135a13c2716427ffc845402046dc"}
Nov 02 23:32:21.136266 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.136185    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs"
Nov 02 23:32:21.136266 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.136217    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs" event={"ID":"6b218d62-c9f7-46d6-b1fc-23a20569ecd7","Type":"ContainerDied","Data":"ceb415e8f97cff28f70cfe2290212ac95994ab9c339bedb09efb1f826b9fff84"}
Nov 02 23:32:21.144512 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.144484    4294 scope.go:122] "RemoveContainer" containerID="489fb1578c48645e4d33e402c136dd875cadadd1e34a93c1131f36c76c6bcb1e"
Nov 02 23:32:21.144808 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:32:21.144781    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489fb1578c48645e4d33e402c136dd875cadadd1e34a93c1131f36c76c6bcb1e\": container with ID starting with 489fb1578c48645e4d33e402c136dd875cadadd1e34a93c1131f36c76c6bcb1e not found: ID does not exist" containerID="489fb1578c48645e4d33e402c136dd875cadadd1e34a93c1131f36c76c6bcb1e"
Nov 02 23:32:21.144899 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.144806    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489fb1578c48645e4d33e402c136dd875cadadd1e34a93c1131f36c76c6bcb1e"} err="failed to get container status \"489fb1578c48645e4d33e402c136dd875cadadd1e34a93c1131f36c76c6bcb1e\": rpc error: code = NotFound desc = could not find container \"489fb1578c48645e4d33e402c136dd875cadadd1e34a93c1131f36c76c6bcb1e\": container with ID starting with 489fb1578c48645e4d33e402c136dd875cadadd1e34a93c1131f36c76c6bcb1e not found: ID does not exist"
Nov 02 23:32:21.144899 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.144823    4294 scope.go:122] "RemoveContainer" containerID="9ac3b5f6a4580e49da35f6d7ef18a06a5d21135a13c2716427ffc845402046dc"
Nov 02 23:32:21.153771 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.153753    4294 scope.go:122] "RemoveContainer" containerID="9ac3b5f6a4580e49da35f6d7ef18a06a5d21135a13c2716427ffc845402046dc"
Nov 02 23:32:21.154030 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:32:21.154007    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac3b5f6a4580e49da35f6d7ef18a06a5d21135a13c2716427ffc845402046dc\": container with ID starting with 9ac3b5f6a4580e49da35f6d7ef18a06a5d21135a13c2716427ffc845402046dc not found: ID does not exist" containerID="9ac3b5f6a4580e49da35f6d7ef18a06a5d21135a13c2716427ffc845402046dc"
Nov 02 23:32:21.154097 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.154037    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac3b5f6a4580e49da35f6d7ef18a06a5d21135a13c2716427ffc845402046dc"} err="failed to get container status \"9ac3b5f6a4580e49da35f6d7ef18a06a5d21135a13c2716427ffc845402046dc\": rpc error: code = NotFound desc = could not find container \"9ac3b5f6a4580e49da35f6d7ef18a06a5d21135a13c2716427ffc845402046dc\": container with ID starting with 9ac3b5f6a4580e49da35f6d7ef18a06a5d21135a13c2716427ffc845402046dc not found: ID does not exist"
Nov 02 23:32:21.209000 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.208930    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsg22\" (UniqueName: \"kubernetes.io/projected/c9c7b45d-5085-430a-aec1-d4d09eb35e52-kube-api-access-qsg22\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:32:21.209000 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.208956    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c7b45d-5085-430a-aec1-d4d09eb35e52-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:32:21.330878 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.330843    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs"]
Nov 02 23:32:21.334940 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.334916    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-02cdd9-69b69bd8f-lpwrs"]
Nov 02 23:32:21.440094 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.440064    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl"]
Nov 02 23:32:21.444365 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.444333    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-02cdd9-59d74f9dbc-gq6zl"]
Nov 02 23:32:21.891698 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.891667    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf"]
Nov 02 23:32:21.892235 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.892221    4294 cpu_manager.go:414] "containerMap: removing container" podUID="6b218d62-c9f7-46d6-b1fc-23a20569ecd7" containerName="payload-processing"
Nov 02 23:32:21.892235 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.892236    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b218d62-c9f7-46d6-b1fc-23a20569ecd7" containerName="payload-processing"
Nov 02 23:32:21.892318 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.892249    4294 cpu_manager.go:414] "containerMap: removing container" podUID="c9c7b45d-5085-430a-aec1-d4d09eb35e52" containerName="payload-pre-processing"
Nov 02 23:32:21.892318 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.892254    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c7b45d-5085-430a-aec1-d4d09eb35e52" containerName="payload-pre-processing"
Nov 02 23:32:21.892378 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.892338    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b218d62-c9f7-46d6-b1fc-23a20569ecd7" containerName="payload-processing"
Nov 02 23:32:21.892378 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.892347    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c7b45d-5085-430a-aec1-d4d09eb35e52" containerName="payload-pre-processing"
Nov 02 23:32:21.896960 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.896939    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf"
Nov 02 23:32:21.921837 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.921812    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a163ef-e0ea-4087-ab54-937b1a98ce6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf\" (UID: \"93a163ef-e0ea-4087-ab54-937b1a98ce6c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf"
Nov 02 23:32:21.922002 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.921960    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np4hr\" (UniqueName: \"kubernetes.io/projected/93a163ef-e0ea-4087-ab54-937b1a98ce6c-kube-api-access-np4hr\") pod \"payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf\" (UID: \"93a163ef-e0ea-4087-ab54-937b1a98ce6c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf"
Nov 02 23:32:21.949076 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.947290    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-0fac41\""
Nov 02 23:32:21.949076 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.948672    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf"]
Nov 02 23:32:21.991315 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:21.991276    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-0fac41-dockercfg-27mt7\""
Nov 02 23:32:22.023251 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.023087    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np4hr\" (UniqueName: \"kubernetes.io/projected/93a163ef-e0ea-4087-ab54-937b1a98ce6c-kube-api-access-np4hr\") pod \"payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf\" (UID: \"93a163ef-e0ea-4087-ab54-937b1a98ce6c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf"
Nov 02 23:32:22.023251 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.023202    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a163ef-e0ea-4087-ab54-937b1a98ce6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf\" (UID: \"93a163ef-e0ea-4087-ab54-937b1a98ce6c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf"
Nov 02 23:32:22.023962 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.023944    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a163ef-e0ea-4087-ab54-937b1a98ce6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf\" (UID: \"93a163ef-e0ea-4087-ab54-937b1a98ce6c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf"
Nov 02 23:32:22.043063 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.043038    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5"]
Nov 02 23:32:22.046900 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.046884    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5"
Nov 02 23:32:22.077591 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.077564    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np4hr\" (UniqueName: \"kubernetes.io/projected/93a163ef-e0ea-4087-ab54-937b1a98ce6c-kube-api-access-np4hr\") pod \"payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf\" (UID: \"93a163ef-e0ea-4087-ab54-937b1a98ce6c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf"
Nov 02 23:32:22.102739 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.102711    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5"]
Nov 02 23:32:22.124865 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.124834    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae14430b-e79e-4d76-8bd6-43f107c1940c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5\" (UID: \"ae14430b-e79e-4d76-8bd6-43f107c1940c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5"
Nov 02 23:32:22.124971 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.124949    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbvzd\" (UniqueName: \"kubernetes.io/projected/ae14430b-e79e-4d76-8bd6-43f107c1940c-kube-api-access-cbvzd\") pod \"payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5\" (UID: \"ae14430b-e79e-4d76-8bd6-43f107c1940c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5"
Nov 02 23:32:22.210300 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.210232    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf"
Nov 02 23:32:22.225776 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.225757    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae14430b-e79e-4d76-8bd6-43f107c1940c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5\" (UID: \"ae14430b-e79e-4d76-8bd6-43f107c1940c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5"
Nov 02 23:32:22.225852 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.225810    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbvzd\" (UniqueName: \"kubernetes.io/projected/ae14430b-e79e-4d76-8bd6-43f107c1940c-kube-api-access-cbvzd\") pod \"payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5\" (UID: \"ae14430b-e79e-4d76-8bd6-43f107c1940c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5"
Nov 02 23:32:22.226443 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.226423    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae14430b-e79e-4d76-8bd6-43f107c1940c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5\" (UID: \"ae14430b-e79e-4d76-8bd6-43f107c1940c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5"
Nov 02 23:32:22.281555 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.281522    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbvzd\" (UniqueName: \"kubernetes.io/projected/ae14430b-e79e-4d76-8bd6-43f107c1940c-kube-api-access-cbvzd\") pod \"payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5\" (UID: \"ae14430b-e79e-4d76-8bd6-43f107c1940c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5"
Nov 02 23:32:22.361820 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.361770    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5"
Nov 02 23:32:22.547878 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.547848    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b218d62-c9f7-46d6-b1fc-23a20569ecd7" path="/var/lib/kubelet/pods/6b218d62-c9f7-46d6-b1fc-23a20569ecd7/volumes"
Nov 02 23:32:22.548339 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.548322    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c7b45d-5085-430a-aec1-d4d09eb35e52" path="/var/lib/kubelet/pods/c9c7b45d-5085-430a-aec1-d4d09eb35e52/volumes"
Nov 02 23:32:22.574125 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.574098    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf"]
Nov 02 23:32:22.574310 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:32:22.574291    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a163ef_e0ea_4087_ab54_937b1a98ce6c.slice/crio-bf72dfd9763d9eb475636cb2d4bc3afb7a92d1d190594bd0ebe46194ec005521 WatchSource:0}: Error finding container bf72dfd9763d9eb475636cb2d4bc3afb7a92d1d190594bd0ebe46194ec005521: Status 404 returned error can't find the container with id bf72dfd9763d9eb475636cb2d4bc3afb7a92d1d190594bd0ebe46194ec005521
Nov 02 23:32:22.698309 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.698258    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:22.728060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.728030    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5"]
Nov 02 23:32:22.728179 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:32:22.728113    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae14430b_e79e_4d76_8bd6_43f107c1940c.slice/crio-11cbed44edf18687edd52711aba03663fcc99c40aea0e776db26ffab52f4832f WatchSource:0}: Error finding container 11cbed44edf18687edd52711aba03663fcc99c40aea0e776db26ffab52f4832f: Status 404 returned error can't find the container with id 11cbed44edf18687edd52711aba03663fcc99c40aea0e776db26ffab52f4832f
Nov 02 23:32:22.848207 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:22.848124    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:23.151761 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:23.151660    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf" event={"ID":"93a163ef-e0ea-4087-ab54-937b1a98ce6c","Type":"ContainerStarted","Data":"2b848cb4a2866e5b237fd8ec62609143444b0abbb41ff2dfe5ed4aaf497ff95c"}
Nov 02 23:32:23.151761 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:23.151740    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf" event={"ID":"93a163ef-e0ea-4087-ab54-937b1a98ce6c","Type":"ContainerStarted","Data":"bf72dfd9763d9eb475636cb2d4bc3afb7a92d1d190594bd0ebe46194ec005521"}
Nov 02 23:32:23.152308 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:23.151879    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf"
Nov 02 23:32:23.153359 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:23.153338    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5" event={"ID":"ae14430b-e79e-4d76-8bd6-43f107c1940c","Type":"ContainerStarted","Data":"9069b1845f7309a940c5a2468a612c36e154c7042089e4e242c156307b7f48ab"}
Nov 02 23:32:23.153423 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:23.153375    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5" event={"ID":"ae14430b-e79e-4d76-8bd6-43f107c1940c","Type":"ContainerStarted","Data":"11cbed44edf18687edd52711aba03663fcc99c40aea0e776db26ffab52f4832f"}
Nov 02 23:32:23.153462 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:23.153428    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5"
Nov 02 23:32:23.263542 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:23.263484    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf" podStartSLOduration=2.263472146 podStartE2EDuration="2.263472146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:23.260261447 +0000 UTC m=+2382.828616248" watchObservedRunningTime="2026-11-02 23:32:23.263472146 +0000 UTC m=+2382.831826933"
Nov 02 23:32:23.371364 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:23.371307    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5" podStartSLOduration=1.371292655 podStartE2EDuration="1.371292655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:32:23.368681879 +0000 UTC m=+2382.937036666" watchObservedRunningTime="2026-11-02 23:32:23.371292655 +0000 UTC m=+2382.939647442"
Nov 02 23:32:27.112206 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:27.112130    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h"
Nov 02 23:32:27.112577 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:27.112530    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r"
Nov 02 23:32:27.185437 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:27.185403    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:27.355697 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:27.355665    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:27.740798 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:27.740766    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:28.806281 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:28.806246    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:29.162157 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:29.162078    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"]
Nov 02 23:32:29.162324 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:29.162285    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f" podUID="7f6e952c-2543-4730-b9f3-598d55c51d0d" containerName="istio-proxy" containerID="cri-o://0a927e3faaf5ec7932b28558559309c72676f755fa042299e386e8b0db03203c" gracePeriod=30
Nov 02 23:32:29.196439 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:29.196412    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:33.034116 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:33.034080    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:34.163492 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.163454    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5"
Nov 02 23:32:34.163909 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.163622    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf"
Nov 02 23:32:34.436687 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.436619    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:32:34.570661 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.570627    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-data\") pod \"7f6e952c-2543-4730-b9f3-598d55c51d0d\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") "
Nov 02 23:32:34.570846 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.570675    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-token\" (UniqueName: \"kubernetes.io/projected/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-token\") pod \"7f6e952c-2543-4730-b9f3-598d55c51d0d\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") "
Nov 02 23:32:34.570846 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.570698    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-workload-socket\") pod \"7f6e952c-2543-4730-b9f3-598d55c51d0d\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") "
Nov 02 23:32:34.570846 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.570729    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-workload-certs\") pod \"7f6e952c-2543-4730-b9f3-598d55c51d0d\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") "
Nov 02 23:32:34.570846 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.570797    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-podinfo\") pod \"7f6e952c-2543-4730-b9f3-598d55c51d0d\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") "
Nov 02 23:32:34.571162 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.571139    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-envoy\") pod \"7f6e952c-2543-4730-b9f3-598d55c51d0d\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") "
Nov 02 23:32:34.571252 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.570922    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-data" pod "7f6e952c-2543-4730-b9f3-598d55c51d0d" (UID: "7f6e952c-2543-4730-b9f3-598d55c51d0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:32:34.571252 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.571186    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-credential-socket\") pod \"7f6e952c-2543-4730-b9f3-598d55c51d0d\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") "
Nov 02 23:32:34.571252 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.571077    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-workload-socket" pod "7f6e952c-2543-4730-b9f3-598d55c51d0d" (UID: "7f6e952c-2543-4730-b9f3-598d55c51d0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:32:34.571252 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.571098    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-workload-certs" pod "7f6e952c-2543-4730-b9f3-598d55c51d0d" (UID: "7f6e952c-2543-4730-b9f3-598d55c51d0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:32:34.571496 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.571291    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f6e952c-2543-4730-b9f3-598d55c51d0d-kube-api-access-2pq95\" (UniqueName: \"kubernetes.io/projected/7f6e952c-2543-4730-b9f3-598d55c51d0d-kube-api-access-2pq95\") pod \"7f6e952c-2543-4730-b9f3-598d55c51d0d\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") "
Nov 02 23:32:34.571496 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.571323    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f6e952c-2543-4730-b9f3-598d55c51d0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f6e952c-2543-4730-b9f3-598d55c51d0d-istiod-ca-cert\") pod \"7f6e952c-2543-4730-b9f3-598d55c51d0d\" (UID: \"7f6e952c-2543-4730-b9f3-598d55c51d0d\") "
Nov 02 23:32:34.571496 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.571468    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-credential-socket" pod "7f6e952c-2543-4730-b9f3-598d55c51d0d" (UID: "7f6e952c-2543-4730-b9f3-598d55c51d0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:32:34.572070 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.572046    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:32:34.572190 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.572087    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:32:34.572190 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.572104    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:32:34.572190 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.572117    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:32:34.572190 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.572152    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f6e952c-2543-4730-b9f3-598d55c51d0d-istiod-ca-cert" pod "7f6e952c-2543-4730-b9f3-598d55c51d0d" (UID: "7f6e952c-2543-4730-b9f3-598d55c51d0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:32:34.573185 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.573151    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-podinfo" pod "7f6e952c-2543-4730-b9f3-598d55c51d0d" (UID: "7f6e952c-2543-4730-b9f3-598d55c51d0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:32:34.573427 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.573404    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-token" pod "7f6e952c-2543-4730-b9f3-598d55c51d0d" (UID: "7f6e952c-2543-4730-b9f3-598d55c51d0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:32:34.573526 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.573477    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-envoy" pod "7f6e952c-2543-4730-b9f3-598d55c51d0d" (UID: "7f6e952c-2543-4730-b9f3-598d55c51d0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:32:34.574170 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.574154    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f6e952c-2543-4730-b9f3-598d55c51d0d-kube-api-access-2pq95" pod "7f6e952c-2543-4730-b9f3-598d55c51d0d" (UID: "7f6e952c-2543-4730-b9f3-598d55c51d0d"). InnerVolumeSpecName "kube-api-access-2pq95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:32:34.673569 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.673530    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:32:34.673569 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.673561    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:32:34.673569 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.673574    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pq95\" (UniqueName: \"kubernetes.io/projected/7f6e952c-2543-4730-b9f3-598d55c51d0d-kube-api-access-2pq95\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:32:34.673815 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.673658    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f6e952c-2543-4730-b9f3-598d55c51d0d-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:32:34.673815 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.673671    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f6e952c-2543-4730-b9f3-598d55c51d0d-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:32:34.888258 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:34.888223    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:35.225348 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:35.225312    4294 generic.go:356] "Generic (PLEG): container finished" podID="7f6e952c-2543-4730-b9f3-598d55c51d0d" containerID="0a927e3faaf5ec7932b28558559309c72676f755fa042299e386e8b0db03203c" exitCode=0
Nov 02 23:32:35.225833 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:35.225365    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"
Nov 02 23:32:35.225833 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:35.225405    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f" event={"ID":"7f6e952c-2543-4730-b9f3-598d55c51d0d","Type":"ContainerDied","Data":"0a927e3faaf5ec7932b28558559309c72676f755fa042299e386e8b0db03203c"}
Nov 02 23:32:35.225833 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:35.225470    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f" event={"ID":"7f6e952c-2543-4730-b9f3-598d55c51d0d","Type":"ContainerDied","Data":"e473a20072cebb386e1cf0698f5d843bcd8e5f1ae32915fea937f160460dbbb1"}
Nov 02 23:32:35.225833 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:35.225504    4294 scope.go:122] "RemoveContainer" containerID="0a927e3faaf5ec7932b28558559309c72676f755fa042299e386e8b0db03203c"
Nov 02 23:32:35.235814 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:35.235795    4294 scope.go:122] "RemoveContainer" containerID="4168679baef5fe0ab3ad49f409530fa8296d7f92283cc2ec83440b469a791767"
Nov 02 23:32:35.244283 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:35.244265    4294 scope.go:122] "RemoveContainer" containerID="0a927e3faaf5ec7932b28558559309c72676f755fa042299e386e8b0db03203c"
Nov 02 23:32:35.244564 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:32:35.244546    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a927e3faaf5ec7932b28558559309c72676f755fa042299e386e8b0db03203c\": container with ID starting with 0a927e3faaf5ec7932b28558559309c72676f755fa042299e386e8b0db03203c not found: ID does not exist" containerID="0a927e3faaf5ec7932b28558559309c72676f755fa042299e386e8b0db03203c"
Nov 02 23:32:35.244603 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:35.244574    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a927e3faaf5ec7932b28558559309c72676f755fa042299e386e8b0db03203c"} err="failed to get container status \"0a927e3faaf5ec7932b28558559309c72676f755fa042299e386e8b0db03203c\": rpc error: code = NotFound desc = could not find container \"0a927e3faaf5ec7932b28558559309c72676f755fa042299e386e8b0db03203c\": container with ID starting with 0a927e3faaf5ec7932b28558559309c72676f755fa042299e386e8b0db03203c not found: ID does not exist"
Nov 02 23:32:35.244603 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:35.244593    4294 scope.go:122] "RemoveContainer" containerID="4168679baef5fe0ab3ad49f409530fa8296d7f92283cc2ec83440b469a791767"
Nov 02 23:32:35.244844 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:32:35.244827    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4168679baef5fe0ab3ad49f409530fa8296d7f92283cc2ec83440b469a791767\": container with ID starting with 4168679baef5fe0ab3ad49f409530fa8296d7f92283cc2ec83440b469a791767 not found: ID does not exist" containerID="4168679baef5fe0ab3ad49f409530fa8296d7f92283cc2ec83440b469a791767"
Nov 02 23:32:35.244889 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:35.244848    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4168679baef5fe0ab3ad49f409530fa8296d7f92283cc2ec83440b469a791767"} err="failed to get container status \"4168679baef5fe0ab3ad49f409530fa8296d7f92283cc2ec83440b469a791767\": rpc error: code = NotFound desc = could not find container \"4168679baef5fe0ab3ad49f409530fa8296d7f92283cc2ec83440b469a791767\": container with ID starting with 4168679baef5fe0ab3ad49f409530fa8296d7f92283cc2ec83440b469a791767 not found: ID does not exist"
Nov 02 23:32:35.432732 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:35.432655    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"]
Nov 02 23:32:35.480330 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:35.480293    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-02cdd9-openshift-default-76bf4cdf7c-sx65f"]
Nov 02 23:32:36.548455 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:36.548423    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f6e952c-2543-4730-b9f3-598d55c51d0d" path="/var/lib/kubelet/pods/7f6e952c-2543-4730-b9f3-598d55c51d0d/volumes"
Nov 02 23:32:37.995350 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:37.995315    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:41.752998 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:32:41.752953    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:32:43.007379 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:43.007333    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:49.591263 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:49.591216    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:54.474208 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:54.474141    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:32:59.454797 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:32:59.454734    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:04.533592 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:04.533559    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:08.161315 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:08.161284    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:08.668055 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:08.668022    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:10.515948 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:10.515914    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg"]
Nov 02 23:33:10.516520 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:10.516503    4294 cpu_manager.go:414] "containerMap: removing container" podUID="7f6e952c-2543-4730-b9f3-598d55c51d0d" containerName="istio-proxy"
Nov 02 23:33:10.516564 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:10.516521    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f6e952c-2543-4730-b9f3-598d55c51d0d" containerName="istio-proxy"
Nov 02 23:33:10.516564 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:10.516546    4294 cpu_manager.go:414] "containerMap: removing container" podUID="7f6e952c-2543-4730-b9f3-598d55c51d0d" containerName="istio-proxy"
Nov 02 23:33:10.516564 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:10.516551    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f6e952c-2543-4730-b9f3-598d55c51d0d" containerName="istio-proxy"
Nov 02 23:33:10.516668 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:10.516659    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f6e952c-2543-4730-b9f3-598d55c51d0d" containerName="istio-proxy"
Nov 02 23:33:10.516708 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:10.516673    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f6e952c-2543-4730-b9f3-598d55c51d0d" containerName="istio-proxy"
Nov 02 23:33:10.519970 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:10.519956    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg"
Nov 02 23:33:10.572871 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:10.572848    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg"]
Nov 02 23:33:10.585056 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:10.585033    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5a3909-dockercfg-6zc4h\""
Nov 02 23:33:10.616173 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:10.616152    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5a3909\""
Nov 02 23:33:10.676819 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:10.676777    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf9cp\" (UniqueName: \"kubernetes.io/projected/bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b-kube-api-access-xf9cp\") pod \"payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg\" (UID: \"bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg"
Nov 02 23:33:10.677029 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:10.677006    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg\" (UID: \"bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg"
Nov 02 23:33:10.778414 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:10.778345    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf9cp\" (UniqueName: \"kubernetes.io/projected/bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b-kube-api-access-xf9cp\") pod \"payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg\" (UID: \"bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg"
Nov 02 23:33:10.778414 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:10.778400    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg\" (UID: \"bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg"
Nov 02 23:33:10.779119 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:10.779094    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg\" (UID: \"bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg"
Nov 02 23:33:10.832879 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:10.832852    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf9cp\" (UniqueName: \"kubernetes.io/projected/bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b-kube-api-access-xf9cp\") pod \"payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg\" (UID: \"bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg"
Nov 02 23:33:10.833659 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:10.833643    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg"
Nov 02 23:33:11.197570 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:11.197545    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg"]
Nov 02 23:33:11.197679 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:33:11.197607    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf80a0ba_f0f9_4651_b6aa_def6c34c7c7b.slice/crio-4c4bf7296979b7f5c53a449661ddf96c739d3011bce7d4cfab940317430478c4 WatchSource:0}: Error finding container 4c4bf7296979b7f5c53a449661ddf96c739d3011bce7d4cfab940317430478c4: Status 404 returned error can't find the container with id 4c4bf7296979b7f5c53a449661ddf96c739d3011bce7d4cfab940317430478c4
Nov 02 23:33:11.334600 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:11.334568    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:11.437110 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:11.437078    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg" event={"ID":"bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b","Type":"ContainerStarted","Data":"50dad91fadc174d6206fc98b7ff35b9fd63d17ba836df0e7f74a3250080edb6e"}
Nov 02 23:33:11.437290 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:11.437134    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg" event={"ID":"bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b","Type":"ContainerStarted","Data":"4c4bf7296979b7f5c53a449661ddf96c739d3011bce7d4cfab940317430478c4"}
Nov 02 23:33:11.437290 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:11.437282    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg"
Nov 02 23:33:11.548519 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:11.548472    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg" podStartSLOduration=1.548458589 podStartE2EDuration="1.548458589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:33:11.54551521 +0000 UTC m=+2431.113869997" watchObservedRunningTime="2026-11-02 23:33:11.548458589 +0000 UTC m=+2431.116813372"
Nov 02 23:33:11.752834 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:33:11.752795    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:33:14.045298 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:14.045262    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:16.193085 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:16.193015    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:18.966247 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:18.966212    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"]
Nov 02 23:33:19.062862 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:19.062831    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:20.427573 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:20.427540    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:20.586172 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:20.586141    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:21.064313 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:21.064264    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:21.233352 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:21.233293    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:21.390377 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:21.390299    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:22.448741 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:22.448712    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg"
Nov 02 23:33:26.302881 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:26.302844    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:32.500536 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:32.500504    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"]
Nov 02 23:33:32.504872 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:32.504849    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"
Nov 02 23:33:32.559115 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:32.559082    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"]
Nov 02 23:33:32.602665 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:32.602630    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-0fac41-serving-cert\""
Nov 02 23:33:32.640042 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:32.639972    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-metrics-certs\") pod \"maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b\" (UID: \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"
Nov 02 23:33:32.640234 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:32.640111    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b\" (UID: \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"
Nov 02 23:33:32.640234 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:32.640145    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b\" (UID: \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"
Nov 02 23:33:32.640336 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:32.640242    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9srs8\" (UniqueName: \"kubernetes.io/projected/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-kube-api-access-9srs8\") pod \"maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b\" (UID: \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"
Nov 02 23:33:32.731509 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:32.731473    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:32.742068 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:32.742044    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b\" (UID: \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"
Nov 02 23:33:32.742204 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:32.742111    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9srs8\" (UniqueName: \"kubernetes.io/projected/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-kube-api-access-9srs8\") pod \"maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b\" (UID: \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"
Nov 02 23:33:32.742263 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:33:32.742197    4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-serving-cert: secret "maas-api-e2e-praxis-b-0fac41-serving-cert" not found
Nov 02 23:33:32.742302 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:33:32.742269    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-maas-api-tls podName:bbc5c3d4-e68e-4988-9750-28fa49d1e05f nodeName:}" failed. No retries permitted until 2026-11-02 23:33:33.242252257 +0000 UTC m=+2452.810607025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-maas-api-tls") pod "maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b" (UID: "bbc5c3d4-e68e-4988-9750-28fa49d1e05f") : secret "maas-api-e2e-praxis-b-0fac41-serving-cert" not found
Nov 02 23:33:32.742302 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:32.742291    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-metrics-certs\") pod \"maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b\" (UID: \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"
Nov 02 23:33:32.742407 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:32.742383    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b\" (UID: \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"
Nov 02 23:33:32.743035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:32.743015    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b\" (UID: \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"
Nov 02 23:33:32.745130 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:32.745107    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-metrics-certs\") pod \"maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b\" (UID: \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"
Nov 02 23:33:32.799293 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:32.799219    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9srs8\" (UniqueName: \"kubernetes.io/projected/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-kube-api-access-9srs8\") pod \"maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b\" (UID: \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"
Nov 02 23:33:32.888835 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:32.888790    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:33.077429 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:33.077348    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:33.248362 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:33.248332    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b\" (UID: \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"
Nov 02 23:33:33.250706 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:33.250681    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-maas-api-tls\") pod \"maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b\" (UID: \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"
Nov 02 23:33:33.418795 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:33.418721    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"
Nov 02 23:33:33.788536 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:33.788505    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"]
Nov 02 23:33:33.789157 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:33:33.789132    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc5c3d4_e68e_4988_9750_28fa49d1e05f.slice/crio-90fd9dd889e98f657b25e890e37e79ca0dfeb88e8808f1ee3fe37e425e083b56 WatchSource:0}: Error finding container 90fd9dd889e98f657b25e890e37e79ca0dfeb88e8808f1ee3fe37e425e083b56: Status 404 returned error can't find the container with id 90fd9dd889e98f657b25e890e37e79ca0dfeb88e8808f1ee3fe37e425e083b56
Nov 02 23:33:33.790764 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:33.790744    4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 23:33:33.987238 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:33.987201    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh" podUID="e0141f50-637d-4e3f-af89-02424a5aa910" containerName="main" containerID="cri-o://d6b90580849dd61b26172c4d520628f5c5051f32f2cd3ec2eddc1732c9d01449" gracePeriod=45
Nov 02 23:33:34.253924 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.253898    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:33:34.361735 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.361659    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0141f50-637d-4e3f-af89-02424a5aa910-kube-api-access-zzmbx\" (UniqueName: \"kubernetes.io/projected/e0141f50-637d-4e3f-af89-02424a5aa910-kube-api-access-zzmbx\") pod \"e0141f50-637d-4e3f-af89-02424a5aa910\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") "
Nov 02 23:33:34.361735 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.361731    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0141f50-637d-4e3f-af89-02424a5aa910-tls-certs\" (UniqueName: \"kubernetes.io/secret/e0141f50-637d-4e3f-af89-02424a5aa910-tls-certs\") pod \"e0141f50-637d-4e3f-af89-02424a5aa910\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") "
Nov 02 23:33:34.361960 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.361819    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-dshm\") pod \"e0141f50-637d-4e3f-af89-02424a5aa910\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") "
Nov 02 23:33:34.361960 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.361840    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-model-cache\") pod \"e0141f50-637d-4e3f-af89-02424a5aa910\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") "
Nov 02 23:33:34.361960 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.361882    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-tmp-dir\") pod \"e0141f50-637d-4e3f-af89-02424a5aa910\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") "
Nov 02 23:33:34.361960 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.361935    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-home\" (UniqueName: \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-home\") pod \"e0141f50-637d-4e3f-af89-02424a5aa910\" (UID: \"e0141f50-637d-4e3f-af89-02424a5aa910\") "
Nov 02 23:33:34.362225 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.362140    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-model-cache" pod "e0141f50-637d-4e3f-af89-02424a5aa910" (UID: "e0141f50-637d-4e3f-af89-02424a5aa910"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:33:34.362225 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.362203    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-tmp-dir" pod "e0141f50-637d-4e3f-af89-02424a5aa910" (UID: "e0141f50-637d-4e3f-af89-02424a5aa910"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:33:34.362225 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.362214    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-home" pod "e0141f50-637d-4e3f-af89-02424a5aa910" (UID: "e0141f50-637d-4e3f-af89-02424a5aa910"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:33:34.362500 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.362481    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:33:34.362500 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.362500    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:33:34.362642 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.362509    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:33:34.364230 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.364209    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0141f50-637d-4e3f-af89-02424a5aa910-kube-api-access-zzmbx" pod "e0141f50-637d-4e3f-af89-02424a5aa910" (UID: "e0141f50-637d-4e3f-af89-02424a5aa910"). InnerVolumeSpecName "kube-api-access-zzmbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:33:34.364404 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.364383    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0141f50-637d-4e3f-af89-02424a5aa910-tls-certs" pod "e0141f50-637d-4e3f-af89-02424a5aa910" (UID: "e0141f50-637d-4e3f-af89-02424a5aa910"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:33:34.364450 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.364383    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-dshm" pod "e0141f50-637d-4e3f-af89-02424a5aa910" (UID: "e0141f50-637d-4e3f-af89-02424a5aa910"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:33:34.466958 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.466909    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzmbx\" (UniqueName: \"kubernetes.io/projected/e0141f50-637d-4e3f-af89-02424a5aa910-kube-api-access-zzmbx\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:33:34.466958 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.466957    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0141f50-637d-4e3f-af89-02424a5aa910-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:33:34.467232 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.466973    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0141f50-637d-4e3f-af89-02424a5aa910-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:33:34.576400 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.576371    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b" event={"ID":"bbc5c3d4-e68e-4988-9750-28fa49d1e05f","Type":"ContainerStarted","Data":"90fd9dd889e98f657b25e890e37e79ca0dfeb88e8808f1ee3fe37e425e083b56"}
Nov 02 23:33:34.577538 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.577514    4294 generic.go:356] "Generic (PLEG): container finished" podID="e0141f50-637d-4e3f-af89-02424a5aa910" containerID="d6b90580849dd61b26172c4d520628f5c5051f32f2cd3ec2eddc1732c9d01449" exitCode=0
Nov 02 23:33:34.577642 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.577565    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"
Nov 02 23:33:34.577642 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.577592    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh" event={"ID":"e0141f50-637d-4e3f-af89-02424a5aa910","Type":"ContainerDied","Data":"d6b90580849dd61b26172c4d520628f5c5051f32f2cd3ec2eddc1732c9d01449"}
Nov 02 23:33:34.577642 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.577628    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh" event={"ID":"e0141f50-637d-4e3f-af89-02424a5aa910","Type":"ContainerDied","Data":"3e4212681892f4e091656c852b37b9665f259bd85f63feb69e55337498f0c229"}
Nov 02 23:33:34.577752 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.577644    4294 scope.go:122] "RemoveContainer" containerID="d6b90580849dd61b26172c4d520628f5c5051f32f2cd3ec2eddc1732c9d01449"
Nov 02 23:33:34.587159 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.587136    4294 scope.go:122] "RemoveContainer" containerID="d6b90580849dd61b26172c4d520628f5c5051f32f2cd3ec2eddc1732c9d01449"
Nov 02 23:33:34.587447 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:33:34.587427    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b90580849dd61b26172c4d520628f5c5051f32f2cd3ec2eddc1732c9d01449\": container with ID starting with d6b90580849dd61b26172c4d520628f5c5051f32f2cd3ec2eddc1732c9d01449 not found: ID does not exist" containerID="d6b90580849dd61b26172c4d520628f5c5051f32f2cd3ec2eddc1732c9d01449"
Nov 02 23:33:34.587499 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.587455    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b90580849dd61b26172c4d520628f5c5051f32f2cd3ec2eddc1732c9d01449"} err="failed to get container status \"d6b90580849dd61b26172c4d520628f5c5051f32f2cd3ec2eddc1732c9d01449\": rpc error: code = NotFound desc = could not find container \"d6b90580849dd61b26172c4d520628f5c5051f32f2cd3ec2eddc1732c9d01449\": container with ID starting with d6b90580849dd61b26172c4d520628f5c5051f32f2cd3ec2eddc1732c9d01449 not found: ID does not exist"
Nov 02 23:33:34.738031 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.738001    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"]
Nov 02 23:33:34.768418 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:34.768381    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-81c3c9/praxis-route-81c3c9-kserve-58bd655467-fp9kh"]
Nov 02 23:33:35.227670 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:35.227636    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:35.392838 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:35.392754    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:35.586644 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:35.586602    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b" event={"ID":"bbc5c3d4-e68e-4988-9750-28fa49d1e05f","Type":"ContainerStarted","Data":"f9fad49ba7fc362b6f9529eb63d9d3c20f81b357ecfbca9efb8061f121d0d2e3"}
Nov 02 23:33:35.587084 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:35.586787    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"
Nov 02 23:33:35.700353 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:35.700262    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b" podStartSLOduration=2.826249005 podStartE2EDuration="3.700247899s" totalImagesPullingTime="873.998894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:33:35.697419058 +0000 UTC m=+2455.265773845" watchObservedRunningTime="2026-11-02 23:33:35.700247899 +0000 UTC m=+2455.268602721"
Nov 02 23:33:36.548431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:36.548397    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0141f50-637d-4e3f-af89-02424a5aa910" path="/var/lib/kubelet/pods/e0141f50-637d-4e3f-af89-02424a5aa910/volumes"
Nov 02 23:33:37.901370 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:37.901336    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:41.600046 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:41.600020    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"
Nov 02 23:33:41.754194 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:33:41.754140    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:33:42.775940 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:42.775907    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:47.682510 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:47.682478    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:52.587698 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:52.587667    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:55.313011 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:55.312920    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:56.345433 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:56.345397    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:56.510839 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:56.510805    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:56.677178 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:56.677105    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:33:57.826909 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:33:57.826874    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:01.908343 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:01.908314    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:04.920604 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:04.920571    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:05.193241 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:05.193173    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rnnmk_cab95f3e-f51d-45b6-a218-e0c272a09aac/payload-processing/0.log"
Nov 02 23:34:05.436471 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:05.436444    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rnnmk_cab95f3e-f51d-45b6-a218-e0c272a09aac/payload-processing/0.log"
Nov 02 23:34:06.835415 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:06.835385    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:07.569773 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:07.569739    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:09.886542 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:09.886510    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:11.753918 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:34:11.753880    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:34:11.760708 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:11.760684    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"]
Nov 02 23:34:11.860532 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:11.860498    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:15.000141 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:15.000109    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:20.065947 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:20.065914    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:26.781407 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:26.781357    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx" podUID="2c2acb51-b739-4375-a332-13837baa1c01" containerName="main" containerID="cri-o://ccc3653907efe3dae9b62d319e3b7ca0c8a8053f331be9e794999b2e99cedf62" gracePeriod=45
Nov 02 23:34:26.894699 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:26.894656    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx" podUID="2c2acb51-b739-4375-a332-13837baa1c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": dial tcp 10.133.0.119:8000: connect: connection refused"
Nov 02 23:34:26.941462 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:26.941428    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:27.054178 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.054155    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:34:27.155898 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.155862    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-tmp-dir\") pod \"2c2acb51-b739-4375-a332-13837baa1c01\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") "
Nov 02 23:34:27.156138 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.155943    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-dshm\") pod \"2c2acb51-b739-4375-a332-13837baa1c01\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") "
Nov 02 23:34:27.156138 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.155974    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-model-cache\") pod \"2c2acb51-b739-4375-a332-13837baa1c01\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") "
Nov 02 23:34:27.156271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.156149    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c2acb51-b739-4375-a332-13837baa1c01-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c2acb51-b739-4375-a332-13837baa1c01-tls-certs\") pod \"2c2acb51-b739-4375-a332-13837baa1c01\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") "
Nov 02 23:34:27.156271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.156193    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c2acb51-b739-4375-a332-13837baa1c01-kube-api-access-rpfdf\" (UniqueName: \"kubernetes.io/projected/2c2acb51-b739-4375-a332-13837baa1c01-kube-api-access-rpfdf\") pod \"2c2acb51-b739-4375-a332-13837baa1c01\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") "
Nov 02 23:34:27.156271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.156223    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-home\" (UniqueName: \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-home\") pod \"2c2acb51-b739-4375-a332-13837baa1c01\" (UID: \"2c2acb51-b739-4375-a332-13837baa1c01\") "
Nov 02 23:34:27.156271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.156226    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-tmp-dir" pod "2c2acb51-b739-4375-a332-13837baa1c01" (UID: "2c2acb51-b739-4375-a332-13837baa1c01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:34:27.156468 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.156398    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-model-cache" pod "2c2acb51-b739-4375-a332-13837baa1c01" (UID: "2c2acb51-b739-4375-a332-13837baa1c01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:34:27.156843 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.156777    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:34:27.156843 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.156790    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-home" pod "2c2acb51-b739-4375-a332-13837baa1c01" (UID: "2c2acb51-b739-4375-a332-13837baa1c01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:34:27.156843 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.156800    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:34:27.158701 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.158672    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c2acb51-b739-4375-a332-13837baa1c01-tls-certs" pod "2c2acb51-b739-4375-a332-13837baa1c01" (UID: "2c2acb51-b739-4375-a332-13837baa1c01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:34:27.158821 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.158779    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2acb51-b739-4375-a332-13837baa1c01-kube-api-access-rpfdf" pod "2c2acb51-b739-4375-a332-13837baa1c01" (UID: "2c2acb51-b739-4375-a332-13837baa1c01"). InnerVolumeSpecName "kube-api-access-rpfdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:34:27.158879 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.158812    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-dshm" pod "2c2acb51-b739-4375-a332-13837baa1c01" (UID: "2c2acb51-b739-4375-a332-13837baa1c01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:34:27.258346 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.258315    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c2acb51-b739-4375-a332-13837baa1c01-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:34:27.258346 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.258344    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpfdf\" (UniqueName: \"kubernetes.io/projected/2c2acb51-b739-4375-a332-13837baa1c01-kube-api-access-rpfdf\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:34:27.258549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.258356    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:34:27.258549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.258365    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c2acb51-b739-4375-a332-13837baa1c01-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:34:27.903904 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.903872    4294 generic.go:356] "Generic (PLEG): container finished" podID="2c2acb51-b739-4375-a332-13837baa1c01" containerID="ccc3653907efe3dae9b62d319e3b7ca0c8a8053f331be9e794999b2e99cedf62" exitCode=0
Nov 02 23:34:27.904475 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.903925    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx" event={"ID":"2c2acb51-b739-4375-a332-13837baa1c01","Type":"ContainerDied","Data":"ccc3653907efe3dae9b62d319e3b7ca0c8a8053f331be9e794999b2e99cedf62"}
Nov 02 23:34:27.904475 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.903936    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"
Nov 02 23:34:27.904475 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.903962    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx" event={"ID":"2c2acb51-b739-4375-a332-13837baa1c01","Type":"ContainerDied","Data":"24ae6dc1267e7fe9b736a9c04085524d785c6948de7ad180d67f3330cc28af64"}
Nov 02 23:34:27.904475 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.903994    4294 scope.go:122] "RemoveContainer" containerID="ccc3653907efe3dae9b62d319e3b7ca0c8a8053f331be9e794999b2e99cedf62"
Nov 02 23:34:27.916242 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.916191    4294 scope.go:122] "RemoveContainer" containerID="ccc3653907efe3dae9b62d319e3b7ca0c8a8053f331be9e794999b2e99cedf62"
Nov 02 23:34:27.916614 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:34:27.916584    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccc3653907efe3dae9b62d319e3b7ca0c8a8053f331be9e794999b2e99cedf62\": container with ID starting with ccc3653907efe3dae9b62d319e3b7ca0c8a8053f331be9e794999b2e99cedf62 not found: ID does not exist" containerID="ccc3653907efe3dae9b62d319e3b7ca0c8a8053f331be9e794999b2e99cedf62"
Nov 02 23:34:27.916704 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:27.916624    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccc3653907efe3dae9b62d319e3b7ca0c8a8053f331be9e794999b2e99cedf62"} err="failed to get container status \"ccc3653907efe3dae9b62d319e3b7ca0c8a8053f331be9e794999b2e99cedf62\": rpc error: code = NotFound desc = could not find container \"ccc3653907efe3dae9b62d319e3b7ca0c8a8053f331be9e794999b2e99cedf62\": container with ID starting with ccc3653907efe3dae9b62d319e3b7ca0c8a8053f331be9e794999b2e99cedf62 not found: ID does not exist"
Nov 02 23:34:28.064529 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:28.064494    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"]
Nov 02 23:34:28.114018 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:28.113958    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-981724/ipp-route-981724-kserve-7dd9464db5-f85hx"]
Nov 02 23:34:28.548314 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:28.548278    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c2acb51-b739-4375-a332-13837baa1c01" path="/var/lib/kubelet/pods/2c2acb51-b739-4375-a332-13837baa1c01/volumes"
Nov 02 23:34:31.784812 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:31.784778    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:35.556899 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:35.556861    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:36.670938 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:36.670893    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:41.750154 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:34:41.750116    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:34:41.788561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:41.788510    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:44.376785 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:44.376733    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:45.139236 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:45.139203    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h"]
Nov 02 23:34:45.245339 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:45.245299    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r"]
Nov 02 23:34:45.543118 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:45.543081    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:45.729834 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:45.729797    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:48.061538 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:48.061509    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rnnmk_cab95f3e-f51d-45b6-a218-e0c272a09aac/payload-processing/0.log"
Nov 02 23:34:48.306893 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:48.306867    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rnnmk_cab95f3e-f51d-45b6-a218-e0c272a09aac/payload-processing/0.log"
Nov 02 23:34:48.365355 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:48.364554    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:48.804198 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:48.804168    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:49.272452 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:49.272421    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:49.737445 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:49.737347    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:50.140551 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:50.140511    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h" podUID="8402d643-1fb6-4d39-862f-f702b0f685a2" containerName="payload-processing" containerID="cri-o://13a947fc667e45b6c54498593c52b5d9e51bec381f61fcf0079ba62634e26dd6" gracePeriod=25
Nov 02 23:34:50.246557 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:50.246516    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r" podUID="cbedb5c3-3260-41bb-ba5e-50d6e518b458" containerName="payload-pre-processing" containerID="cri-o://4f0ee58e7e7d853f43ed1c13432dcde18f46ca33cb0b75340e72624f3dac9d68" gracePeriod=25
Nov 02 23:34:50.425743 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:50.425716    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h"
Nov 02 23:34:50.513285 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:50.513257    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r"
Nov 02 23:34:50.513459 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:50.513438    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8402d643-1fb6-4d39-862f-f702b0f685a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8402d643-1fb6-4d39-862f-f702b0f685a2-plugins-config-volume\") pod \"8402d643-1fb6-4d39-862f-f702b0f685a2\" (UID: \"8402d643-1fb6-4d39-862f-f702b0f685a2\") "
Nov 02 23:34:50.513621 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:50.513607    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8402d643-1fb6-4d39-862f-f702b0f685a2-kube-api-access-pqq58\" (UniqueName: \"kubernetes.io/projected/8402d643-1fb6-4d39-862f-f702b0f685a2-kube-api-access-pqq58\") pod \"8402d643-1fb6-4d39-862f-f702b0f685a2\" (UID: \"8402d643-1fb6-4d39-862f-f702b0f685a2\") "
Nov 02 23:34:50.513765 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:50.513749    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8402d643-1fb6-4d39-862f-f702b0f685a2-plugins-config-volume" pod "8402d643-1fb6-4d39-862f-f702b0f685a2" (UID: "8402d643-1fb6-4d39-862f-f702b0f685a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:34:50.514031 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:50.514012    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8402d643-1fb6-4d39-862f-f702b0f685a2-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:34:50.515503 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:50.515485    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8402d643-1fb6-4d39-862f-f702b0f685a2-kube-api-access-pqq58" pod "8402d643-1fb6-4d39-862f-f702b0f685a2" (UID: "8402d643-1fb6-4d39-862f-f702b0f685a2"). InnerVolumeSpecName "kube-api-access-pqq58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:34:50.615392 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:50.615350    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbedb5c3-3260-41bb-ba5e-50d6e518b458-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbedb5c3-3260-41bb-ba5e-50d6e518b458-plugins-config-volume\") pod \"cbedb5c3-3260-41bb-ba5e-50d6e518b458\" (UID: \"cbedb5c3-3260-41bb-ba5e-50d6e518b458\") "
Nov 02 23:34:50.615567 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:50.615507    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbedb5c3-3260-41bb-ba5e-50d6e518b458-kube-api-access-q8q7s\" (UniqueName: \"kubernetes.io/projected/cbedb5c3-3260-41bb-ba5e-50d6e518b458-kube-api-access-q8q7s\") pod \"cbedb5c3-3260-41bb-ba5e-50d6e518b458\" (UID: \"cbedb5c3-3260-41bb-ba5e-50d6e518b458\") "
Nov 02 23:34:50.615738 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:50.615715    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbedb5c3-3260-41bb-ba5e-50d6e518b458-plugins-config-volume" pod "cbedb5c3-3260-41bb-ba5e-50d6e518b458" (UID: "cbedb5c3-3260-41bb-ba5e-50d6e518b458"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:34:50.616090 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:50.616071    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbedb5c3-3260-41bb-ba5e-50d6e518b458-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:34:50.616090 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:50.616091    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqq58\" (UniqueName: \"kubernetes.io/projected/8402d643-1fb6-4d39-862f-f702b0f685a2-kube-api-access-pqq58\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:34:50.617640 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:50.617622    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbedb5c3-3260-41bb-ba5e-50d6e518b458-kube-api-access-q8q7s" pod "cbedb5c3-3260-41bb-ba5e-50d6e518b458" (UID: "cbedb5c3-3260-41bb-ba5e-50d6e518b458"). InnerVolumeSpecName "kube-api-access-q8q7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:34:50.717105 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:50.717026    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8q7s\" (UniqueName: \"kubernetes.io/projected/cbedb5c3-3260-41bb-ba5e-50d6e518b458-kube-api-access-q8q7s\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:34:51.043712 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:51.043680    4294 generic.go:356] "Generic (PLEG): container finished" podID="cbedb5c3-3260-41bb-ba5e-50d6e518b458" containerID="4f0ee58e7e7d853f43ed1c13432dcde18f46ca33cb0b75340e72624f3dac9d68" exitCode=0
Nov 02 23:34:51.043922 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:51.043735    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r"
Nov 02 23:34:51.043922 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:51.043759    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r" event={"ID":"cbedb5c3-3260-41bb-ba5e-50d6e518b458","Type":"ContainerDied","Data":"4f0ee58e7e7d853f43ed1c13432dcde18f46ca33cb0b75340e72624f3dac9d68"}
Nov 02 23:34:51.043922 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:51.043813    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r" event={"ID":"cbedb5c3-3260-41bb-ba5e-50d6e518b458","Type":"ContainerDied","Data":"99a683287cf50aa713ff299610203b2d0bcbc7d2748f50a20efbc3b9d31c8786"}
Nov 02 23:34:51.043922 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:51.043838    4294 scope.go:122] "RemoveContainer" containerID="4f0ee58e7e7d853f43ed1c13432dcde18f46ca33cb0b75340e72624f3dac9d68"
Nov 02 23:34:51.045157 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:51.045123    4294 generic.go:356] "Generic (PLEG): container finished" podID="8402d643-1fb6-4d39-862f-f702b0f685a2" containerID="13a947fc667e45b6c54498593c52b5d9e51bec381f61fcf0079ba62634e26dd6" exitCode=0
Nov 02 23:34:51.045287 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:51.045169    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h" event={"ID":"8402d643-1fb6-4d39-862f-f702b0f685a2","Type":"ContainerDied","Data":"13a947fc667e45b6c54498593c52b5d9e51bec381f61fcf0079ba62634e26dd6"}
Nov 02 23:34:51.045287 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:51.045221    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h" event={"ID":"8402d643-1fb6-4d39-862f-f702b0f685a2","Type":"ContainerDied","Data":"3334f8f2f27b378e7a903ec279fea0f46f17f406937d800303e605a47d59ac12"}
Nov 02 23:34:51.045287 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:51.045172    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h"
Nov 02 23:34:51.054644 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:51.054624    4294 scope.go:122] "RemoveContainer" containerID="4f0ee58e7e7d853f43ed1c13432dcde18f46ca33cb0b75340e72624f3dac9d68"
Nov 02 23:34:51.054940 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:34:51.054923    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0ee58e7e7d853f43ed1c13432dcde18f46ca33cb0b75340e72624f3dac9d68\": container with ID starting with 4f0ee58e7e7d853f43ed1c13432dcde18f46ca33cb0b75340e72624f3dac9d68 not found: ID does not exist" containerID="4f0ee58e7e7d853f43ed1c13432dcde18f46ca33cb0b75340e72624f3dac9d68"
Nov 02 23:34:51.055009 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:51.054947    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0ee58e7e7d853f43ed1c13432dcde18f46ca33cb0b75340e72624f3dac9d68"} err="failed to get container status \"4f0ee58e7e7d853f43ed1c13432dcde18f46ca33cb0b75340e72624f3dac9d68\": rpc error: code = NotFound desc = could not find container \"4f0ee58e7e7d853f43ed1c13432dcde18f46ca33cb0b75340e72624f3dac9d68\": container with ID starting with 4f0ee58e7e7d853f43ed1c13432dcde18f46ca33cb0b75340e72624f3dac9d68 not found: ID does not exist"
Nov 02 23:34:51.055009 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:51.054964    4294 scope.go:122] "RemoveContainer" containerID="13a947fc667e45b6c54498593c52b5d9e51bec381f61fcf0079ba62634e26dd6"
Nov 02 23:34:51.064210 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:51.064150    4294 scope.go:122] "RemoveContainer" containerID="13a947fc667e45b6c54498593c52b5d9e51bec381f61fcf0079ba62634e26dd6"
Nov 02 23:34:51.064470 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:34:51.064449    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a947fc667e45b6c54498593c52b5d9e51bec381f61fcf0079ba62634e26dd6\": container with ID starting with 13a947fc667e45b6c54498593c52b5d9e51bec381f61fcf0079ba62634e26dd6 not found: ID does not exist" containerID="13a947fc667e45b6c54498593c52b5d9e51bec381f61fcf0079ba62634e26dd6"
Nov 02 23:34:51.064510 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:51.064477    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a947fc667e45b6c54498593c52b5d9e51bec381f61fcf0079ba62634e26dd6"} err="failed to get container status \"13a947fc667e45b6c54498593c52b5d9e51bec381f61fcf0079ba62634e26dd6\": rpc error: code = NotFound desc = could not find container \"13a947fc667e45b6c54498593c52b5d9e51bec381f61fcf0079ba62634e26dd6\": container with ID starting with 13a947fc667e45b6c54498593c52b5d9e51bec381f61fcf0079ba62634e26dd6 not found: ID does not exist"
Nov 02 23:34:51.201250 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:51.201217    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r"]
Nov 02 23:34:51.248347 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:51.248313    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db5d87-69cb746c87-kjk5r"]
Nov 02 23:34:51.307258 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:51.307165    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h"]
Nov 02 23:34:51.309961 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:51.309934    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-db5d87-8c4687bbc-s8c5h"]
Nov 02 23:34:52.548805 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:52.548771    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8402d643-1fb6-4d39-862f-f702b0f685a2" path="/var/lib/kubelet/pods/8402d643-1fb6-4d39-862f-f702b0f685a2/volumes"
Nov 02 23:34:52.549218 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:52.549176    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbedb5c3-3260-41bb-ba5e-50d6e518b458" path="/var/lib/kubelet/pods/cbedb5c3-3260-41bb-ba5e-50d6e518b458/volumes"
Nov 02 23:34:54.192098 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:54.192058    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:55.399230 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:55.399200    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:55.768893 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:55.768844    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:56.134599 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:56.134476    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:58.170161 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:58.170112    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:34:59.542356 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:34:59.542325    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:02.144703 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:02.144666    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:04.120228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:04.120192    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:04.517277 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:04.517241    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:04.886529 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:04.886452    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:05.067148 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.067113    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"]
Nov 02 23:35:05.067711 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.067696    4294 cpu_manager.go:414] "containerMap: removing container" podUID="cbedb5c3-3260-41bb-ba5e-50d6e518b458" containerName="payload-pre-processing"
Nov 02 23:35:05.067772 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.067713    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbedb5c3-3260-41bb-ba5e-50d6e518b458" containerName="payload-pre-processing"
Nov 02 23:35:05.067772 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.067736    4294 cpu_manager.go:414] "containerMap: removing container" podUID="e0141f50-637d-4e3f-af89-02424a5aa910" containerName="main"
Nov 02 23:35:05.067772 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.067741    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0141f50-637d-4e3f-af89-02424a5aa910" containerName="main"
Nov 02 23:35:05.067772 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.067767    4294 cpu_manager.go:414] "containerMap: removing container" podUID="2c2acb51-b739-4375-a332-13837baa1c01" containerName="main"
Nov 02 23:35:05.067772 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.067773    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c2acb51-b739-4375-a332-13837baa1c01" containerName="main"
Nov 02 23:35:05.067935 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.067780    4294 cpu_manager.go:414] "containerMap: removing container" podUID="8402d643-1fb6-4d39-862f-f702b0f685a2" containerName="payload-processing"
Nov 02 23:35:05.067935 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.067785    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8402d643-1fb6-4d39-862f-f702b0f685a2" containerName="payload-processing"
Nov 02 23:35:05.067935 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.067872    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c2acb51-b739-4375-a332-13837baa1c01" containerName="main"
Nov 02 23:35:05.067935 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.067882    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8402d643-1fb6-4d39-862f-f702b0f685a2" containerName="payload-processing"
Nov 02 23:35:05.067935 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.067891    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbedb5c3-3260-41bb-ba5e-50d6e518b458" containerName="payload-pre-processing"
Nov 02 23:35:05.067935 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.067898    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0141f50-637d-4e3f-af89-02424a5aa910" containerName="main"
Nov 02 23:35:05.072960 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.072939    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.076775 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.076753    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:05.175651 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.175572    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"]
Nov 02 23:35:05.187042 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.187016    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.187209 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.187050    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.187209 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.187087    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.187330 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.187213    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.187330 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.187268    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.187454 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.187382    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmrr\" (UniqueName: \"kubernetes.io/projected/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-kube-api-access-ghmrr\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.187454 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.187427    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.187560 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.187478    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.187560 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.187522    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.288849 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.288817    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.288849 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.288854    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.289109 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.288902    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.289109 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.288945    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.289109 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.288974    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.289109 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.289040    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.289109 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.289092    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.289384 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.289155    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.289384 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.289203    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmrr\" (UniqueName: \"kubernetes.io/projected/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-kube-api-access-ghmrr\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.289384 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.289326    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.289384 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.289337    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.289384 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.289345    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.289652 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.289584    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.289917 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.289891    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.291614 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.291581    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.292224 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.292200    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.343827 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.343802    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.344061 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.344044    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmrr\" (UniqueName: \"kubernetes.io/projected/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-kube-api-access-ghmrr\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.386831 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.386805    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:05.803120 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.803096    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"]
Nov 02 23:35:05.803262 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:35:05.803222    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec3e6ec_079a_485e_a7f3_3f830c7b89ec.slice/crio-e506a6e7cb651e8bd7ccb5c24019c697e468f5f5d4eb555c6d9faf5e2fc154fc WatchSource:0}: Error finding container e506a6e7cb651e8bd7ccb5c24019c697e468f5f5d4eb555c6d9faf5e2fc154fc: Status 404 returned error can't find the container with id e506a6e7cb651e8bd7ccb5c24019c697e468f5f5d4eb555c6d9faf5e2fc154fc
Nov 02 23:35:05.805645 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.805613    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:35:05.805741 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.805675    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:35:05.805741 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:05.805703    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:35:06.141718 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:06.141620    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c" event={"ID":"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec","Type":"ContainerStarted","Data":"577421ce91774a38149c9e439ced871fe093be2e26c17776e876023d5be45146"}
Nov 02 23:35:06.141718 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:06.141688    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c" event={"ID":"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec","Type":"ContainerStarted","Data":"e506a6e7cb651e8bd7ccb5c24019c697e468f5f5d4eb555c6d9faf5e2fc154fc"}
Nov 02 23:35:06.256222 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:06.256168    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c" podStartSLOduration=1.256152668 podStartE2EDuration="1.256152668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:35:06.253589868 +0000 UTC m=+2545.821944655" watchObservedRunningTime="2026-11-02 23:35:06.256152668 +0000 UTC m=+2545.824507455"
Nov 02 23:35:06.387404 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:06.387375    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:06.389107 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:06.389078    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 02 23:35:06.389231 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:06.389128    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c" podUID="2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 02 23:35:07.387485 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:07.387442    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 02 23:35:07.387955 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:07.387508    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c" podUID="2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 02 23:35:08.388022 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:08.387962    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 02 23:35:08.388422 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:08.388056    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c" podUID="2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 02 23:35:09.387496 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:09.387462    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 02 23:35:09.387650 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:09.387515    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c" podUID="2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 02 23:35:09.547645 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:09.547614    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:10.113669 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:10.113633    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:10.387507 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:10.387428    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 02 23:35:10.387507 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:10.387483    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c" podUID="2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 02 23:35:11.387900 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:11.387863    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 02 23:35:11.388339 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:11.387919    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c" podUID="2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 02 23:35:11.750255 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:35:11.750218    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:35:12.388218 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:12.388179    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 02 23:35:12.388759 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:12.388240    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c" podUID="2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 02 23:35:13.550651 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:13.550617    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:13.551097 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:13.551025    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:13.551786 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:13.551765    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:35:17.050580 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:17.050535    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:21.984853 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:21.984820    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:26.814717 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:26.814515    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:31.098330 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:31.098294    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:31.418525 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:31.418433    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg"]
Nov 02 23:35:31.419029 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:31.418998    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg" podUID="bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b" containerName="payload-processing" containerID="cri-o://50dad91fadc174d6206fc98b7ff35b9fd63d17ba836df0e7f74a3250080edb6e" gracePeriod=30
Nov 02 23:35:31.604178 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:31.604144    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:31.701134 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:31.701111    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg"
Nov 02 23:35:31.811084 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:31.811050    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b-plugins-config-volume\") pod \"bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b\" (UID: \"bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b\") "
Nov 02 23:35:31.811271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:31.811135    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b-kube-api-access-xf9cp\" (UniqueName: \"kubernetes.io/projected/bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b-kube-api-access-xf9cp\") pod \"bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b\" (UID: \"bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b\") "
Nov 02 23:35:31.811531 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:31.811501    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b-plugins-config-volume" pod "bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b" (UID: "bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:35:31.813959 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:31.813932    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b-kube-api-access-xf9cp" pod "bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b" (UID: "bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b"). InnerVolumeSpecName "kube-api-access-xf9cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:35:31.903723 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:31.903691    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:31.912598 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:31.912578    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf9cp\" (UniqueName: \"kubernetes.io/projected/bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b-kube-api-access-xf9cp\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:35:31.912671 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:31.912603    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:35:32.289392 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:32.289355    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:32.322170 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:32.322137    4294 generic.go:356] "Generic (PLEG): container finished" podID="bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b" containerID="50dad91fadc174d6206fc98b7ff35b9fd63d17ba836df0e7f74a3250080edb6e" exitCode=0
Nov 02 23:35:32.322347 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:32.322185    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg"
Nov 02 23:35:32.322347 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:32.322216    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg" event={"ID":"bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b","Type":"ContainerDied","Data":"50dad91fadc174d6206fc98b7ff35b9fd63d17ba836df0e7f74a3250080edb6e"}
Nov 02 23:35:32.322347 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:32.322276    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg" event={"ID":"bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b","Type":"ContainerDied","Data":"4c4bf7296979b7f5c53a449661ddf96c739d3011bce7d4cfab940317430478c4"}
Nov 02 23:35:32.322347 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:32.322304    4294 scope.go:122] "RemoveContainer" containerID="50dad91fadc174d6206fc98b7ff35b9fd63d17ba836df0e7f74a3250080edb6e"
Nov 02 23:35:32.332321 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:32.332299    4294 scope.go:122] "RemoveContainer" containerID="50dad91fadc174d6206fc98b7ff35b9fd63d17ba836df0e7f74a3250080edb6e"
Nov 02 23:35:32.332584 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:35:32.332566    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50dad91fadc174d6206fc98b7ff35b9fd63d17ba836df0e7f74a3250080edb6e\": container with ID starting with 50dad91fadc174d6206fc98b7ff35b9fd63d17ba836df0e7f74a3250080edb6e not found: ID does not exist" containerID="50dad91fadc174d6206fc98b7ff35b9fd63d17ba836df0e7f74a3250080edb6e"
Nov 02 23:35:32.332626 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:32.332593    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50dad91fadc174d6206fc98b7ff35b9fd63d17ba836df0e7f74a3250080edb6e"} err="failed to get container status \"50dad91fadc174d6206fc98b7ff35b9fd63d17ba836df0e7f74a3250080edb6e\": rpc error: code = NotFound desc = could not find container \"50dad91fadc174d6206fc98b7ff35b9fd63d17ba836df0e7f74a3250080edb6e\": container with ID starting with 50dad91fadc174d6206fc98b7ff35b9fd63d17ba836df0e7f74a3250080edb6e not found: ID does not exist"
Nov 02 23:35:32.479095 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:32.479042    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg"]
Nov 02 23:35:32.482323 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:32.482299    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a3909-5c54c64968-m24rg"]
Nov 02 23:35:32.549678 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:32.549589    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b" path="/var/lib/kubelet/pods/bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b/volumes"
Nov 02 23:35:34.082504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:34.082465    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:35.925037 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:35.925001    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:36.663653 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:36.663620    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:36.884503 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:36.884470    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:38.333066 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:38.333033    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:38.970053 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:38.970018    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:39.486138 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:39.486107    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:39.712174 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:39.712119    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:41.748703 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:35:41.748667    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:35:42.315890 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:42.315861    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:42.537421 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:42.537380    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:43.536851 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:43.536818    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:43.815125 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:43.815045    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:44.142992 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:44.142908    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:44.339444 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:44.339407    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:45.095156 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:45.095119    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:45.754486 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:45.754451    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:47.022685 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:47.022647    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:50.420682 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.420649    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"]
Nov 02 23:35:50.421235 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.421220    4294 cpu_manager.go:414] "containerMap: removing container" podUID="bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b" containerName="payload-processing"
Nov 02 23:35:50.421285 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.421236    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b" containerName="payload-processing"
Nov 02 23:35:50.421333 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.421324    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf80a0ba-f0f9-4651-b6aa-def6c34c7c7b" containerName="payload-processing"
Nov 02 23:35:50.431891 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.431854    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.523611 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.523570    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d078f228-be56-497c-87f6-8e94b080770c-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.523611 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.523619    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.523856 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.523678    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7mpp\" (UniqueName: \"kubernetes.io/projected/d078f228-be56-497c-87f6-8e94b080770c-kube-api-access-d7mpp\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.523856 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.523722    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.523856 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.523750    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.523856 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.523775    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.523856 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.523835    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d078f228-be56-497c-87f6-8e94b080770c-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.524147 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.523955    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d078f228-be56-497c-87f6-8e94b080770c-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.524147 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.524047    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.626656 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.626617    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"]
Nov 02 23:35:50.627396 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.627364    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d078f228-be56-497c-87f6-8e94b080770c-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.627537 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.627428    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.627611 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.627586    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7mpp\" (UniqueName: \"kubernetes.io/projected/d078f228-be56-497c-87f6-8e94b080770c-kube-api-access-d7mpp\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.627699 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.627681    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.627770 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.627737    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.627830 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.627788    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.627830 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.627816    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d078f228-be56-497c-87f6-8e94b080770c-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.627945 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.627901    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d078f228-be56-497c-87f6-8e94b080770c-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.627945 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.627938    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.628234 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.628212    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-credential-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.628312 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.628239    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-istio-data\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.628372 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.628311    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-workload-certs\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.628532 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.628509    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-workload-socket\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.628880 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.628857    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d078f228-be56-497c-87f6-8e94b080770c-istiod-ca-cert\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.630438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.630410    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-istio-envoy\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.631108 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.631087    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d078f228-be56-497c-87f6-8e94b080770c-istio-podinfo\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.681846 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.681765    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d078f228-be56-497c-87f6-8e94b080770c-istio-token\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.682035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.682013    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7mpp\" (UniqueName: \"kubernetes.io/projected/d078f228-be56-497c-87f6-8e94b080770c-kube-api-access-d7mpp\") pod \"e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") " pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:50.744850 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:50.744824    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:51.132394 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:51.132361    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:51.161191 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:51.161159    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"]
Nov 02 23:35:51.194562 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:35:51.194537    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd078f228_be56_497c_87f6_8e94b080770c.slice/crio-9b45939f7629f387d30d1fb3c02b43f5ff9cee9a736740138a23bcdb6c9cd189 WatchSource:0}: Error finding container 9b45939f7629f387d30d1fb3c02b43f5ff9cee9a736740138a23bcdb6c9cd189: Status 404 returned error can't find the container with id 9b45939f7629f387d30d1fb3c02b43f5ff9cee9a736740138a23bcdb6c9cd189
Nov 02 23:35:51.196968 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:51.196935    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:35:51.197066 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:51.197021    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:35:51.197106 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:51.197067    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:35:51.466583 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:51.466503    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr" event={"ID":"d078f228-be56-497c-87f6-8e94b080770c","Type":"ContainerStarted","Data":"66429b844d4597187d38d95933bfc0eacd10d269a218a526444506c0468730d1"}
Nov 02 23:35:51.466583 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:51.466554    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr" event={"ID":"d078f228-be56-497c-87f6-8e94b080770c","Type":"ContainerStarted","Data":"9b45939f7629f387d30d1fb3c02b43f5ff9cee9a736740138a23bcdb6c9cd189"}
Nov 02 23:35:51.581118 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:51.581062    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr" podStartSLOduration=1.581047423 podStartE2EDuration="1.581047423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:35:51.579317872 +0000 UTC m=+2591.147672696" watchObservedRunningTime="2026-11-02 23:35:51.581047423 +0000 UTC m=+2591.149402210"
Nov 02 23:35:51.652006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:51.651960    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:51.746115 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:51.746085    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:51.925262 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:51.925141    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5"]
Nov 02 23:35:52.025500 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:52.025417    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf"]
Nov 02 23:35:52.310283 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:52.310198    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:52.489425 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:52.489386    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:52.746588 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:52.746559    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 02 23:35:52.746774 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:52.746614    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr" podUID="d078f228-be56-497c-87f6-8e94b080770c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 02 23:35:53.602412 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:53.602380    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:53.745715 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:53.745680    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 02 23:35:53.745886 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:53.745739    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr" podUID="d078f228-be56-497c-87f6-8e94b080770c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 02 23:35:53.773000 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:53.771946    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:54.235503 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:54.235469    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:54.745444 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:54.745401    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 02 23:35:54.745891 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:54.745456    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr" podUID="d078f228-be56-497c-87f6-8e94b080770c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 02 23:35:55.745132 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:55.745091    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 02 23:35:55.745319 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:55.745146    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr" podUID="d078f228-be56-497c-87f6-8e94b080770c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 02 23:35:56.682243 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:56.682208    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:56.745624 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:56.745588    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 02 23:35:56.745781 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:56.745647    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr" podUID="d078f228-be56-497c-87f6-8e94b080770c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 02 23:35:56.925708 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:56.925668    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5" podUID="ae14430b-e79e-4d76-8bd6-43f107c1940c" containerName="payload-processing" containerID="cri-o://9069b1845f7309a940c5a2468a612c36e154c7042089e4e242c156307b7f48ab" gracePeriod=25
Nov 02 23:35:57.025924 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.025881    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf" podUID="93a163ef-e0ea-4087-ab54-937b1a98ce6c" containerName="payload-pre-processing" containerID="cri-o://2b848cb4a2866e5b237fd8ec62609143444b0abbb41ff2dfe5ed4aaf497ff95c" gracePeriod=25
Nov 02 23:35:57.227910 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.227885    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5"
Nov 02 23:35:57.323024 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.322990    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae14430b-e79e-4d76-8bd6-43f107c1940c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae14430b-e79e-4d76-8bd6-43f107c1940c-plugins-config-volume\") pod \"ae14430b-e79e-4d76-8bd6-43f107c1940c\" (UID: \"ae14430b-e79e-4d76-8bd6-43f107c1940c\") "
Nov 02 23:35:57.323214 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.323173    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae14430b-e79e-4d76-8bd6-43f107c1940c-kube-api-access-cbvzd\" (UniqueName: \"kubernetes.io/projected/ae14430b-e79e-4d76-8bd6-43f107c1940c-kube-api-access-cbvzd\") pod \"ae14430b-e79e-4d76-8bd6-43f107c1940c\" (UID: \"ae14430b-e79e-4d76-8bd6-43f107c1940c\") "
Nov 02 23:35:57.323423 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.323318    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae14430b-e79e-4d76-8bd6-43f107c1940c-plugins-config-volume" pod "ae14430b-e79e-4d76-8bd6-43f107c1940c" (UID: "ae14430b-e79e-4d76-8bd6-43f107c1940c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:35:57.323694 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.323673    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae14430b-e79e-4d76-8bd6-43f107c1940c-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:35:57.325278 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.325255    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae14430b-e79e-4d76-8bd6-43f107c1940c-kube-api-access-cbvzd" pod "ae14430b-e79e-4d76-8bd6-43f107c1940c" (UID: "ae14430b-e79e-4d76-8bd6-43f107c1940c"). InnerVolumeSpecName "kube-api-access-cbvzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:35:57.347842 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.347819    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf"
Nov 02 23:35:57.424173 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.424137    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a163ef-e0ea-4087-ab54-937b1a98ce6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a163ef-e0ea-4087-ab54-937b1a98ce6c-plugins-config-volume\") pod \"93a163ef-e0ea-4087-ab54-937b1a98ce6c\" (UID: \"93a163ef-e0ea-4087-ab54-937b1a98ce6c\") "
Nov 02 23:35:57.424342 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.424207    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a163ef-e0ea-4087-ab54-937b1a98ce6c-kube-api-access-np4hr\" (UniqueName: \"kubernetes.io/projected/93a163ef-e0ea-4087-ab54-937b1a98ce6c-kube-api-access-np4hr\") pod \"93a163ef-e0ea-4087-ab54-937b1a98ce6c\" (UID: \"93a163ef-e0ea-4087-ab54-937b1a98ce6c\") "
Nov 02 23:35:57.424501 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.424479    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a163ef-e0ea-4087-ab54-937b1a98ce6c-plugins-config-volume" pod "93a163ef-e0ea-4087-ab54-937b1a98ce6c" (UID: "93a163ef-e0ea-4087-ab54-937b1a98ce6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:35:57.424657 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.424645    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbvzd\" (UniqueName: \"kubernetes.io/projected/ae14430b-e79e-4d76-8bd6-43f107c1940c-kube-api-access-cbvzd\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:35:57.424708 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.424660    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a163ef-e0ea-4087-ab54-937b1a98ce6c-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:35:57.426229 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.426207    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a163ef-e0ea-4087-ab54-937b1a98ce6c-kube-api-access-np4hr" pod "93a163ef-e0ea-4087-ab54-937b1a98ce6c" (UID: "93a163ef-e0ea-4087-ab54-937b1a98ce6c"). InnerVolumeSpecName "kube-api-access-np4hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:35:57.514865 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.514835    4294 generic.go:356] "Generic (PLEG): container finished" podID="93a163ef-e0ea-4087-ab54-937b1a98ce6c" containerID="2b848cb4a2866e5b237fd8ec62609143444b0abbb41ff2dfe5ed4aaf497ff95c" exitCode=0
Nov 02 23:35:57.515092 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.514887    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf"
Nov 02 23:35:57.515092 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.514919    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf" event={"ID":"93a163ef-e0ea-4087-ab54-937b1a98ce6c","Type":"ContainerDied","Data":"2b848cb4a2866e5b237fd8ec62609143444b0abbb41ff2dfe5ed4aaf497ff95c"}
Nov 02 23:35:57.515092 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.515003    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf" event={"ID":"93a163ef-e0ea-4087-ab54-937b1a98ce6c","Type":"ContainerDied","Data":"bf72dfd9763d9eb475636cb2d4bc3afb7a92d1d190594bd0ebe46194ec005521"}
Nov 02 23:35:57.515092 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.515037    4294 scope.go:122] "RemoveContainer" containerID="2b848cb4a2866e5b237fd8ec62609143444b0abbb41ff2dfe5ed4aaf497ff95c"
Nov 02 23:35:57.516440 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.516422    4294 generic.go:356] "Generic (PLEG): container finished" podID="ae14430b-e79e-4d76-8bd6-43f107c1940c" containerID="9069b1845f7309a940c5a2468a612c36e154c7042089e4e242c156307b7f48ab" exitCode=0
Nov 02 23:35:57.516551 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.516462    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5" event={"ID":"ae14430b-e79e-4d76-8bd6-43f107c1940c","Type":"ContainerDied","Data":"9069b1845f7309a940c5a2468a612c36e154c7042089e4e242c156307b7f48ab"}
Nov 02 23:35:57.516551 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.516472    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5"
Nov 02 23:35:57.516551 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.516503    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5" event={"ID":"ae14430b-e79e-4d76-8bd6-43f107c1940c","Type":"ContainerDied","Data":"11cbed44edf18687edd52711aba03663fcc99c40aea0e776db26ffab52f4832f"}
Nov 02 23:35:57.524656 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.524638    4294 scope.go:122] "RemoveContainer" containerID="2b848cb4a2866e5b237fd8ec62609143444b0abbb41ff2dfe5ed4aaf497ff95c"
Nov 02 23:35:57.524958 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:35:57.524926    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b848cb4a2866e5b237fd8ec62609143444b0abbb41ff2dfe5ed4aaf497ff95c\": container with ID starting with 2b848cb4a2866e5b237fd8ec62609143444b0abbb41ff2dfe5ed4aaf497ff95c not found: ID does not exist" containerID="2b848cb4a2866e5b237fd8ec62609143444b0abbb41ff2dfe5ed4aaf497ff95c"
Nov 02 23:35:57.525094 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.524968    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b848cb4a2866e5b237fd8ec62609143444b0abbb41ff2dfe5ed4aaf497ff95c"} err="failed to get container status \"2b848cb4a2866e5b237fd8ec62609143444b0abbb41ff2dfe5ed4aaf497ff95c\": rpc error: code = NotFound desc = could not find container \"2b848cb4a2866e5b237fd8ec62609143444b0abbb41ff2dfe5ed4aaf497ff95c\": container with ID starting with 2b848cb4a2866e5b237fd8ec62609143444b0abbb41ff2dfe5ed4aaf497ff95c not found: ID does not exist"
Nov 02 23:35:57.525094 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.525005    4294 scope.go:122] "RemoveContainer" containerID="9069b1845f7309a940c5a2468a612c36e154c7042089e4e242c156307b7f48ab"
Nov 02 23:35:57.525437 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.525419    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np4hr\" (UniqueName: \"kubernetes.io/projected/93a163ef-e0ea-4087-ab54-937b1a98ce6c-kube-api-access-np4hr\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:35:57.533539 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.533520    4294 scope.go:122] "RemoveContainer" containerID="9069b1845f7309a940c5a2468a612c36e154c7042089e4e242c156307b7f48ab"
Nov 02 23:35:57.533777 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:35:57.533759    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9069b1845f7309a940c5a2468a612c36e154c7042089e4e242c156307b7f48ab\": container with ID starting with 9069b1845f7309a940c5a2468a612c36e154c7042089e4e242c156307b7f48ab not found: ID does not exist" containerID="9069b1845f7309a940c5a2468a612c36e154c7042089e4e242c156307b7f48ab"
Nov 02 23:35:57.533817 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.533787    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9069b1845f7309a940c5a2468a612c36e154c7042089e4e242c156307b7f48ab"} err="failed to get container status \"9069b1845f7309a940c5a2468a612c36e154c7042089e4e242c156307b7f48ab\": rpc error: code = NotFound desc = could not find container \"9069b1845f7309a940c5a2468a612c36e154c7042089e4e242c156307b7f48ab\": container with ID starting with 9069b1845f7309a940c5a2468a612c36e154c7042089e4e242c156307b7f48ab not found: ID does not exist"
Nov 02 23:35:57.674290 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.674253    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf"]
Nov 02 23:35:57.689065 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.689033    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0fac41-76544d7dc-t5wrf"]
Nov 02 23:35:57.745147 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.745113    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 02 23:35:57.745309 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.745170    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr" podUID="d078f228-be56-497c-87f6-8e94b080770c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 02 23:35:57.783410 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.783326    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5"]
Nov 02 23:35:57.789644 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:57.789616    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0fac41-8699c548ff-mcfg5"]
Nov 02 23:35:58.047805 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:58.047699    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:58.217032 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:58.216998    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:58.547939 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:58.547903    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a163ef-e0ea-4087-ab54-937b1a98ce6c" path="/var/lib/kubelet/pods/93a163ef-e0ea-4087-ab54-937b1a98ce6c/volumes"
Nov 02 23:35:58.548401 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:58.548380    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae14430b-e79e-4d76-8bd6-43f107c1940c" path="/var/lib/kubelet/pods/ae14430b-e79e-4d76-8bd6-43f107c1940c/volumes"
Nov 02 23:35:58.919724 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:58.919637    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:58.920215 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:58.920005    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:58.920760 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:58.920741    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:35:59.103143 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:59.103098    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:35:59.621155 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:35:59.621123    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:04.271194 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:04.271159    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:05.334333 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:05.334296    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:06.452767 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:06.452731    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:06.612840 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:06.612781    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:09.521325 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:09.521272    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:11.109879 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:11.109832    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:11.750620 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:36:11.750586    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:36:16.192398 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:16.192361    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:21.104562 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:21.104527    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:25.987647 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:25.987582    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:29.518516 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:29.518475    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:30.884151 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:30.884103    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:35.215481 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:35.215440    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:35.753495 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:35.753446    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:35.927827 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:35.927787    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:36.229539 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.229447    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:36.311383 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.311341    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"]
Nov 02 23:36:36.311566 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.311552    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4" podUID="f3299cc5-a6a4-4cf1-9603-e14541a1d92e" containerName="maas-api" containerID="cri-o://de8fc5d67baaef3aed68d5c6ec77c8b691b3661ccfa0d0189db7a660e08450fa" gracePeriod=30
Nov 02 23:36:36.400461 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.400423    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:36.559154 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.559122    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:36.569078 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.569056    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"
Nov 02 23:36:36.690383 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.690348    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-maas-api-tls\") pod \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\" (UID: \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\") "
Nov 02 23:36:36.690582 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.690417    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-kube-api-access-slrdb\" (UniqueName: \"kubernetes.io/projected/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-kube-api-access-slrdb\") pod \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\" (UID: \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\") "
Nov 02 23:36:36.690582 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.690460    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-metrics-certs\") pod \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\" (UID: \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\") "
Nov 02 23:36:36.690582 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.690510    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-openshift-service-ca\") pod \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\" (UID: \"f3299cc5-a6a4-4cf1-9603-e14541a1d92e\") "
Nov 02 23:36:36.691303 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.691275    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-openshift-service-ca" pod "f3299cc5-a6a4-4cf1-9603-e14541a1d92e" (UID: "f3299cc5-a6a4-4cf1-9603-e14541a1d92e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:36:36.693244 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.693218    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-metrics-certs" pod "f3299cc5-a6a4-4cf1-9603-e14541a1d92e" (UID: "f3299cc5-a6a4-4cf1-9603-e14541a1d92e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:36:36.693404 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.693387    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-kube-api-access-slrdb" pod "f3299cc5-a6a4-4cf1-9603-e14541a1d92e" (UID: "f3299cc5-a6a4-4cf1-9603-e14541a1d92e"). InnerVolumeSpecName "kube-api-access-slrdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:36:36.693660 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.693631    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-maas-api-tls" pod "f3299cc5-a6a4-4cf1-9603-e14541a1d92e" (UID: "f3299cc5-a6a4-4cf1-9603-e14541a1d92e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:36:36.768257 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.768223    4294 generic.go:356] "Generic (PLEG): container finished" podID="f3299cc5-a6a4-4cf1-9603-e14541a1d92e" containerID="de8fc5d67baaef3aed68d5c6ec77c8b691b3661ccfa0d0189db7a660e08450fa" exitCode=0
Nov 02 23:36:36.768456 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.768268    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"
Nov 02 23:36:36.768456 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.768306    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4" event={"ID":"f3299cc5-a6a4-4cf1-9603-e14541a1d92e","Type":"ContainerDied","Data":"de8fc5d67baaef3aed68d5c6ec77c8b691b3661ccfa0d0189db7a660e08450fa"}
Nov 02 23:36:36.768456 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.768358    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4" event={"ID":"f3299cc5-a6a4-4cf1-9603-e14541a1d92e","Type":"ContainerDied","Data":"e3fabda7211a50cad590d809f4cab8777228a20a6d476b4c7ddfe1c5f2ec740b"}
Nov 02 23:36:36.768456 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.768382    4294 scope.go:122] "RemoveContainer" containerID="de8fc5d67baaef3aed68d5c6ec77c8b691b3661ccfa0d0189db7a660e08450fa"
Nov 02 23:36:36.778874 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.778854    4294 scope.go:122] "RemoveContainer" containerID="de8fc5d67baaef3aed68d5c6ec77c8b691b3661ccfa0d0189db7a660e08450fa"
Nov 02 23:36:36.779149 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:36:36.779130    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8fc5d67baaef3aed68d5c6ec77c8b691b3661ccfa0d0189db7a660e08450fa\": container with ID starting with de8fc5d67baaef3aed68d5c6ec77c8b691b3661ccfa0d0189db7a660e08450fa not found: ID does not exist" containerID="de8fc5d67baaef3aed68d5c6ec77c8b691b3661ccfa0d0189db7a660e08450fa"
Nov 02 23:36:36.779212 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.779155    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8fc5d67baaef3aed68d5c6ec77c8b691b3661ccfa0d0189db7a660e08450fa"} err="failed to get container status \"de8fc5d67baaef3aed68d5c6ec77c8b691b3661ccfa0d0189db7a660e08450fa\": rpc error: code = NotFound desc = could not find container \"de8fc5d67baaef3aed68d5c6ec77c8b691b3661ccfa0d0189db7a660e08450fa\": container with ID starting with de8fc5d67baaef3aed68d5c6ec77c8b691b3661ccfa0d0189db7a660e08450fa not found: ID does not exist"
Nov 02 23:36:36.791963 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.791900    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slrdb\" (UniqueName: \"kubernetes.io/projected/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-kube-api-access-slrdb\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:36:36.791963 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.791933    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:36:36.791963 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.791949    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:36:36.792154 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.791963    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3299cc5-a6a4-4cf1-9603-e14541a1d92e-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:36:36.925944 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.925912    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"]
Nov 02 23:36:36.931344 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.931320    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-981724-7567dfbcb4-xnld4"]
Nov 02 23:36:36.963159 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:36.963107    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:37.128527 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:37.128444    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:37.525740 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:37.525707    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"]
Nov 02 23:36:37.526210 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:37.525900    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b" podUID="bbc5c3d4-e68e-4988-9750-28fa49d1e05f" containerName="maas-api" containerID="cri-o://f9fad49ba7fc362b6f9529eb63d9d3c20f81b357ecfbca9efb8061f121d0d2e3" gracePeriod=30
Nov 02 23:36:37.664522 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:37.664493    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:37.776228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:37.775958    4294 generic.go:356] "Generic (PLEG): container finished" podID="bbc5c3d4-e68e-4988-9750-28fa49d1e05f" containerID="f9fad49ba7fc362b6f9529eb63d9d3c20f81b357ecfbca9efb8061f121d0d2e3" exitCode=0
Nov 02 23:36:37.776228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:37.776012    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b" event={"ID":"bbc5c3d4-e68e-4988-9750-28fa49d1e05f","Type":"ContainerDied","Data":"f9fad49ba7fc362b6f9529eb63d9d3c20f81b357ecfbca9efb8061f121d0d2e3"}
Nov 02 23:36:37.776228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:37.776076    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b" event={"ID":"bbc5c3d4-e68e-4988-9750-28fa49d1e05f","Type":"ContainerDied","Data":"90fd9dd889e98f657b25e890e37e79ca0dfeb88e8808f1ee3fe37e425e083b56"}
Nov 02 23:36:37.776228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:37.776101    4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90fd9dd889e98f657b25e890e37e79ca0dfeb88e8808f1ee3fe37e425e083b56"
Nov 02 23:36:37.796449 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:37.796424    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"
Nov 02 23:36:37.906703 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:37.906661    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-metrics-certs\") pod \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\" (UID: \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\") "
Nov 02 23:36:37.906905 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:37.906720    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-openshift-service-ca\") pod \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\" (UID: \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\") "
Nov 02 23:36:37.906905 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:37.906792    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-maas-api-tls\") pod \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\" (UID: \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\") "
Nov 02 23:36:37.906905 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:37.906896    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-kube-api-access-9srs8\" (UniqueName: \"kubernetes.io/projected/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-kube-api-access-9srs8\") pod \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\" (UID: \"bbc5c3d4-e68e-4988-9750-28fa49d1e05f\") "
Nov 02 23:36:37.907179 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:37.907156    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-openshift-service-ca" pod "bbc5c3d4-e68e-4988-9750-28fa49d1e05f" (UID: "bbc5c3d4-e68e-4988-9750-28fa49d1e05f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:36:37.907420 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:37.907387    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:36:37.914621 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:37.914558    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-maas-api-tls" pod "bbc5c3d4-e68e-4988-9750-28fa49d1e05f" (UID: "bbc5c3d4-e68e-4988-9750-28fa49d1e05f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:36:37.916228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:37.916201    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-kube-api-access-9srs8" pod "bbc5c3d4-e68e-4988-9750-28fa49d1e05f" (UID: "bbc5c3d4-e68e-4988-9750-28fa49d1e05f"). InnerVolumeSpecName "kube-api-access-9srs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:36:37.929804 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:37.929771    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-metrics-certs" pod "bbc5c3d4-e68e-4988-9750-28fa49d1e05f" (UID: "bbc5c3d4-e68e-4988-9750-28fa49d1e05f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:36:38.008729 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:38.008693    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9srs8\" (UniqueName: \"kubernetes.io/projected/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-kube-api-access-9srs8\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:36:38.008729 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:38.008726    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:36:38.008729 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:38.008735    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbc5c3d4-e68e-4988-9750-28fa49d1e05f-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:36:38.213302 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:38.213206    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:38.555699 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:38.555645    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3299cc5-a6a4-4cf1-9603-e14541a1d92e" path="/var/lib/kubelet/pods/f3299cc5-a6a4-4cf1-9603-e14541a1d92e/volumes"
Nov 02 23:36:38.690097 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:38.690060    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:38.783545 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:38.783515    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"
Nov 02 23:36:38.939880 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:38.939793    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"]
Nov 02 23:36:38.987111 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:38.987077    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0fac41-7bf8ccbd6f-25z2b"]
Nov 02 23:36:39.537815 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:39.537779    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:40.547658 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:40.547628    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbc5c3d4-e68e-4988-9750-28fa49d1e05f" path="/var/lib/kubelet/pods/bbc5c3d4-e68e-4988-9750-28fa49d1e05f/volumes"
Nov 02 23:36:41.752317 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:36:41.752273    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:36:44.662763 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:44.662731    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:44.837629 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:44.837590    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:45.893003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:45.890912    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:46.063005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:46.060850    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:46.351340 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:46.351307    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:46.537200 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:46.537166    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:47.539729 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:47.539698    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:48.092730 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:48.092686    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:48.992437 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:48.992402    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:49.207879 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.207845    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"]
Nov 02 23:36:49.208106 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.208083    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln" podUID="23d8c8d7-67c7-4175-9bcd-d21aed75f9d5" containerName="maas-api" containerID="cri-o://518ef8fa04c609371d2e65ec96f37315de52d6c30356c37222920c012518522a" gracePeriod=30
Nov 02 23:36:49.388444 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.388313    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:49.499576 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.499553    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"
Nov 02 23:36:49.566696 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.566662    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:49.663285 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.663205    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-openshift-service-ca\") pod \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\" (UID: \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\") "
Nov 02 23:36:49.663285 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.663239    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-metrics-certs\") pod \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\" (UID: \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\") "
Nov 02 23:36:49.663285 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.663270    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-kube-api-access-mrnvw\" (UniqueName: \"kubernetes.io/projected/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-kube-api-access-mrnvw\") pod \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\" (UID: \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\") "
Nov 02 23:36:49.663526 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.663475    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-maas-api-tls\") pod \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\" (UID: \"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5\") "
Nov 02 23:36:49.663583 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.663557    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-openshift-service-ca" pod "23d8c8d7-67c7-4175-9bcd-d21aed75f9d5" (UID: "23d8c8d7-67c7-4175-9bcd-d21aed75f9d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:36:49.664358 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.664335    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:36:49.665712 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.665682    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-maas-api-tls" pod "23d8c8d7-67c7-4175-9bcd-d21aed75f9d5" (UID: "23d8c8d7-67c7-4175-9bcd-d21aed75f9d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:36:49.665712 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.665703    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-metrics-certs" pod "23d8c8d7-67c7-4175-9bcd-d21aed75f9d5" (UID: "23d8c8d7-67c7-4175-9bcd-d21aed75f9d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:36:49.666106 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.666089    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-kube-api-access-mrnvw" pod "23d8c8d7-67c7-4175-9bcd-d21aed75f9d5" (UID: "23d8c8d7-67c7-4175-9bcd-d21aed75f9d5"). InnerVolumeSpecName "kube-api-access-mrnvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:36:49.765603 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.765568    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:36:49.765603 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.765601    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrnvw\" (UniqueName: \"kubernetes.io/projected/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-kube-api-access-mrnvw\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:36:49.765864 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.765615    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:36:49.869714 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.869675    4294 generic.go:356] "Generic (PLEG): container finished" podID="23d8c8d7-67c7-4175-9bcd-d21aed75f9d5" containerID="518ef8fa04c609371d2e65ec96f37315de52d6c30356c37222920c012518522a" exitCode=0
Nov 02 23:36:49.869863 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.869725    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"
Nov 02 23:36:49.869863 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.869766    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln" event={"ID":"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5","Type":"ContainerDied","Data":"518ef8fa04c609371d2e65ec96f37315de52d6c30356c37222920c012518522a"}
Nov 02 23:36:49.869863 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.869824    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln" event={"ID":"23d8c8d7-67c7-4175-9bcd-d21aed75f9d5","Type":"ContainerDied","Data":"74ebfaf25351eded17675ed0a667a8faf1f0beed217f135086d295206c344009"}
Nov 02 23:36:49.869863 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.869838    4294 scope.go:122] "RemoveContainer" containerID="518ef8fa04c609371d2e65ec96f37315de52d6c30356c37222920c012518522a"
Nov 02 23:36:49.880802 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.880782    4294 scope.go:122] "RemoveContainer" containerID="518ef8fa04c609371d2e65ec96f37315de52d6c30356c37222920c012518522a"
Nov 02 23:36:49.881116 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:36:49.881083    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"518ef8fa04c609371d2e65ec96f37315de52d6c30356c37222920c012518522a\": container with ID starting with 518ef8fa04c609371d2e65ec96f37315de52d6c30356c37222920c012518522a not found: ID does not exist" containerID="518ef8fa04c609371d2e65ec96f37315de52d6c30356c37222920c012518522a"
Nov 02 23:36:49.881237 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:49.881127    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518ef8fa04c609371d2e65ec96f37315de52d6c30356c37222920c012518522a"} err="failed to get container status \"518ef8fa04c609371d2e65ec96f37315de52d6c30356c37222920c012518522a\": rpc error: code = NotFound desc = could not find container \"518ef8fa04c609371d2e65ec96f37315de52d6c30356c37222920c012518522a\": container with ID starting with 518ef8fa04c609371d2e65ec96f37315de52d6c30356c37222920c012518522a not found: ID does not exist"
Nov 02 23:36:50.028708 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:50.028674    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"]
Nov 02 23:36:50.087123 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:50.087081    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1c14e6-6b54cfdfcd-ccvln"]
Nov 02 23:36:50.128504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:50.128470    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:50.547709 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:50.547678    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d8c8d7-67c7-4175-9bcd-d21aed75f9d5" path="/var/lib/kubelet/pods/23d8c8d7-67c7-4175-9bcd-d21aed75f9d5/volumes"
Nov 02 23:36:50.575574 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:50.575534    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:51.558766 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:51.558723    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:51.882964 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:51.882833    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:52.044298 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:52.044231    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:52.526918 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:52.526880    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:53.571178 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:53.571149    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:54.102510 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:54.102476    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:55.371859 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:55.371826    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:55.549579 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:55.549539    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:55.758153 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:55.758124    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:58.529429 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.529373    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:58.700290 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.700256    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:58.862586 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.862493    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:36:58.922445 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.922416    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58"]
Nov 02 23:36:58.923059 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.923040    4294 cpu_manager.go:414] "containerMap: removing container" podUID="bbc5c3d4-e68e-4988-9750-28fa49d1e05f" containerName="maas-api"
Nov 02 23:36:58.923059 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.923059    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc5c3d4-e68e-4988-9750-28fa49d1e05f" containerName="maas-api"
Nov 02 23:36:58.923181 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.923073    4294 cpu_manager.go:414] "containerMap: removing container" podUID="ae14430b-e79e-4d76-8bd6-43f107c1940c" containerName="payload-processing"
Nov 02 23:36:58.923181 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.923078    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae14430b-e79e-4d76-8bd6-43f107c1940c" containerName="payload-processing"
Nov 02 23:36:58.923181 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.923085    4294 cpu_manager.go:414] "containerMap: removing container" podUID="23d8c8d7-67c7-4175-9bcd-d21aed75f9d5" containerName="maas-api"
Nov 02 23:36:58.923181 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.923089    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d8c8d7-67c7-4175-9bcd-d21aed75f9d5" containerName="maas-api"
Nov 02 23:36:58.923181 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.923109    4294 cpu_manager.go:414] "containerMap: removing container" podUID="f3299cc5-a6a4-4cf1-9603-e14541a1d92e" containerName="maas-api"
Nov 02 23:36:58.923181 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.923114    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3299cc5-a6a4-4cf1-9603-e14541a1d92e" containerName="maas-api"
Nov 02 23:36:58.923181 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.923132    4294 cpu_manager.go:414] "containerMap: removing container" podUID="93a163ef-e0ea-4087-ab54-937b1a98ce6c" containerName="payload-pre-processing"
Nov 02 23:36:58.923181 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.923136    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a163ef-e0ea-4087-ab54-937b1a98ce6c" containerName="payload-pre-processing"
Nov 02 23:36:58.923438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.923215    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc5c3d4-e68e-4988-9750-28fa49d1e05f" containerName="maas-api"
Nov 02 23:36:58.923438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.923223    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae14430b-e79e-4d76-8bd6-43f107c1940c" containerName="payload-processing"
Nov 02 23:36:58.923438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.923228    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d8c8d7-67c7-4175-9bcd-d21aed75f9d5" containerName="maas-api"
Nov 02 23:36:58.923438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.923238    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3299cc5-a6a4-4cf1-9603-e14541a1d92e" containerName="maas-api"
Nov 02 23:36:58.923438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.923247    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a163ef-e0ea-4087-ab54-937b1a98ce6c" containerName="payload-pre-processing"
Nov 02 23:36:58.927908 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.927893    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58"
Nov 02 23:36:58.993508 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.993460    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d4f752c-2b7d-4f18-87a0-44b7315464a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58\" (UID: \"3d4f752c-2b7d-4f18-87a0-44b7315464a4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58"
Nov 02 23:36:58.993858 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:58.993834    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mngfs\" (UniqueName: \"kubernetes.io/projected/3d4f752c-2b7d-4f18-87a0-44b7315464a4-kube-api-access-mngfs\") pod \"payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58\" (UID: \"3d4f752c-2b7d-4f18-87a0-44b7315464a4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58"
Nov 02 23:36:59.001091 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.001043    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58"]
Nov 02 23:36:59.034193 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.034166    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-9672a1\""
Nov 02 23:36:59.034338 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.034233    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-9672a1-dockercfg-925cb\""
Nov 02 23:36:59.073905 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.073880    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft"]
Nov 02 23:36:59.077950 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.077932    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft"
Nov 02 23:36:59.096416 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.096388    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mngfs\" (UniqueName: \"kubernetes.io/projected/3d4f752c-2b7d-4f18-87a0-44b7315464a4-kube-api-access-mngfs\") pod \"payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58\" (UID: \"3d4f752c-2b7d-4f18-87a0-44b7315464a4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58"
Nov 02 23:36:59.096570 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.096551    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d4f752c-2b7d-4f18-87a0-44b7315464a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58\" (UID: \"3d4f752c-2b7d-4f18-87a0-44b7315464a4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58"
Nov 02 23:36:59.097211 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.097192    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d4f752c-2b7d-4f18-87a0-44b7315464a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58\" (UID: \"3d4f752c-2b7d-4f18-87a0-44b7315464a4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58"
Nov 02 23:36:59.131421 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.131345    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft"]
Nov 02 23:36:59.150610 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.150577    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mngfs\" (UniqueName: \"kubernetes.io/projected/3d4f752c-2b7d-4f18-87a0-44b7315464a4-kube-api-access-mngfs\") pod \"payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58\" (UID: \"3d4f752c-2b7d-4f18-87a0-44b7315464a4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58"
Nov 02 23:36:59.197660 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.197624    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89551982-54d3-4006-b91f-8abcc25ed76a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft\" (UID: \"89551982-54d3-4006-b91f-8abcc25ed76a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft"
Nov 02 23:36:59.197794 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.197737    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cxj8\" (UniqueName: \"kubernetes.io/projected/89551982-54d3-4006-b91f-8abcc25ed76a-kube-api-access-8cxj8\") pod \"payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft\" (UID: \"89551982-54d3-4006-b91f-8abcc25ed76a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft"
Nov 02 23:36:59.240572 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.240549    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58"
Nov 02 23:36:59.298894 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.298860    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89551982-54d3-4006-b91f-8abcc25ed76a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft\" (UID: \"89551982-54d3-4006-b91f-8abcc25ed76a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft"
Nov 02 23:36:59.299066 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.299034    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cxj8\" (UniqueName: \"kubernetes.io/projected/89551982-54d3-4006-b91f-8abcc25ed76a-kube-api-access-8cxj8\") pod \"payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft\" (UID: \"89551982-54d3-4006-b91f-8abcc25ed76a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft"
Nov 02 23:36:59.303047 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.302386    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89551982-54d3-4006-b91f-8abcc25ed76a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft\" (UID: \"89551982-54d3-4006-b91f-8abcc25ed76a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft"
Nov 02 23:36:59.353172 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.353140    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cxj8\" (UniqueName: \"kubernetes.io/projected/89551982-54d3-4006-b91f-8abcc25ed76a-kube-api-access-8cxj8\") pod \"payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft\" (UID: \"89551982-54d3-4006-b91f-8abcc25ed76a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft"
Nov 02 23:36:59.391281 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.391224    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft"
Nov 02 23:36:59.606364 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:36:59.606338    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4f752c_2b7d_4f18_87a0_44b7315464a4.slice/crio-fe4b236c86b0fad853d0a6da1744edfbf4e5e34b7ad6653c5f3da16edac7b1ff WatchSource:0}: Error finding container fe4b236c86b0fad853d0a6da1744edfbf4e5e34b7ad6653c5f3da16edac7b1ff: Status 404 returned error can't find the container with id fe4b236c86b0fad853d0a6da1744edfbf4e5e34b7ad6653c5f3da16edac7b1ff
Nov 02 23:36:59.606692 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.606572    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58"]
Nov 02 23:36:59.759007 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.758952    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft"]
Nov 02 23:36:59.760100 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:36:59.760069    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89551982_54d3_4006_b91f_8abcc25ed76a.slice/crio-b00266a723a17b61871a1b3f3403374e9f0ba10e5a129f344b9b7daf008c465b WatchSource:0}: Error finding container b00266a723a17b61871a1b3f3403374e9f0ba10e5a129f344b9b7daf008c465b: Status 404 returned error can't find the container with id b00266a723a17b61871a1b3f3403374e9f0ba10e5a129f344b9b7daf008c465b
Nov 02 23:36:59.930911 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.930814    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft" event={"ID":"89551982-54d3-4006-b91f-8abcc25ed76a","Type":"ContainerStarted","Data":"126dd9d76a5b14151f26451d08776c12e34cdb0cdb0411a0e1c1e2c4e77c47c5"}
Nov 02 23:36:59.931084 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.930916    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft"
Nov 02 23:36:59.931084 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.930946    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft" event={"ID":"89551982-54d3-4006-b91f-8abcc25ed76a","Type":"ContainerStarted","Data":"b00266a723a17b61871a1b3f3403374e9f0ba10e5a129f344b9b7daf008c465b"}
Nov 02 23:36:59.932376 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.932347    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58" event={"ID":"3d4f752c-2b7d-4f18-87a0-44b7315464a4","Type":"ContainerStarted","Data":"515e8bab182d2ad9dfbf424b41ae7503debdc0e8d8e628ae50c8c2acd006f0cc"}
Nov 02 23:36:59.932472 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.932383    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58" event={"ID":"3d4f752c-2b7d-4f18-87a0-44b7315464a4","Type":"ContainerStarted","Data":"fe4b236c86b0fad853d0a6da1744edfbf4e5e34b7ad6653c5f3da16edac7b1ff"}
Nov 02 23:36:59.932520 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:36:59.932479    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58"
Nov 02 23:37:00.040172 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:00.040116    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft" podStartSLOduration=1.040103211 podStartE2EDuration="1.040103211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:37:00.038860191 +0000 UTC m=+2659.607214977" watchObservedRunningTime="2026-11-02 23:37:00.040103211 +0000 UTC m=+2659.608458034"
Nov 02 23:37:00.147467 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:00.147409    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58" podStartSLOduration=2.147394211 podStartE2EDuration="2.147394211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:37:00.146647447 +0000 UTC m=+2659.715002235" watchObservedRunningTime="2026-11-02 23:37:00.147394211 +0000 UTC m=+2659.715749043"
Nov 02 23:37:04.073293 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:04.073260    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:08.991353 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:08.991317    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:10.941154 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:10.941125    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58"
Nov 02 23:37:10.941598 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:10.941286    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft"
Nov 02 23:37:11.755831 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:37:11.755791    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:37:13.933468 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:13.933434    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:18.806214 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:18.806181    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:23.783327 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:23.783295    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:27.903484 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:27.903450    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"]
Nov 02 23:37:27.903928 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:27.903637    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2" podUID="b91d5fc6-8e5d-44e9-be6d-58822e9697bb" containerName="maas-api" containerID="cri-o://32d13b54a9e71044a754f0d440242e178c3a4373a8bfa648e97628a790528072" gracePeriod=30
Nov 02 23:37:27.987853 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:27.987819    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:28.131672 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:28.131639    4294 generic.go:356] "Generic (PLEG): container finished" podID="b91d5fc6-8e5d-44e9-be6d-58822e9697bb" containerID="32d13b54a9e71044a754f0d440242e178c3a4373a8bfa648e97628a790528072" exitCode=0
Nov 02 23:37:28.132014 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:28.131879    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2" event={"ID":"b91d5fc6-8e5d-44e9-be6d-58822e9697bb","Type":"ContainerDied","Data":"32d13b54a9e71044a754f0d440242e178c3a4373a8bfa648e97628a790528072"}
Nov 02 23:37:28.187166 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:28.187141    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"
Nov 02 23:37:28.231260 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:28.231234    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-metrics-certs\") pod \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\" (UID: \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\") "
Nov 02 23:37:28.231419 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:28.231404    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-maas-api-tls\") pod \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\" (UID: \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\") "
Nov 02 23:37:28.231518 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:28.231499    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-openshift-service-ca\") pod \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\" (UID: \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\") "
Nov 02 23:37:28.231648 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:28.231627    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-kube-api-access-nf9t4\" (UniqueName: \"kubernetes.io/projected/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-kube-api-access-nf9t4\") pod \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\" (UID: \"b91d5fc6-8e5d-44e9-be6d-58822e9697bb\") "
Nov 02 23:37:28.231867 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:28.231844    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-openshift-service-ca" pod "b91d5fc6-8e5d-44e9-be6d-58822e9697bb" (UID: "b91d5fc6-8e5d-44e9-be6d-58822e9697bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:37:28.232066 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:28.232051    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:37:28.233398 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:28.233373    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-metrics-certs" pod "b91d5fc6-8e5d-44e9-be6d-58822e9697bb" (UID: "b91d5fc6-8e5d-44e9-be6d-58822e9697bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:37:28.233495 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:28.233408    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-maas-api-tls" pod "b91d5fc6-8e5d-44e9-be6d-58822e9697bb" (UID: "b91d5fc6-8e5d-44e9-be6d-58822e9697bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:37:28.233634 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:28.233618    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-kube-api-access-nf9t4" pod "b91d5fc6-8e5d-44e9-be6d-58822e9697bb" (UID: "b91d5fc6-8e5d-44e9-be6d-58822e9697bb"). InnerVolumeSpecName "kube-api-access-nf9t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:37:28.338458 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:28.333993    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf9t4\" (UniqueName: \"kubernetes.io/projected/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-kube-api-access-nf9t4\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:37:28.338458 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:28.334034    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:37:28.338458 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:28.334050    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91d5fc6-8e5d-44e9-be6d-58822e9697bb-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:37:28.552646 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:28.552617    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:28.718689 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:28.718659    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:29.140383 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:29.140348    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2" event={"ID":"b91d5fc6-8e5d-44e9-be6d-58822e9697bb","Type":"ContainerDied","Data":"30f7bf51774d17f0eb5790d375522a2e19f7d8f64bc5c9d76afe312c5c0ce4ad"}
Nov 02 23:37:29.140888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:29.140403    4294 scope.go:122] "RemoveContainer" containerID="32d13b54a9e71044a754f0d440242e178c3a4373a8bfa648e97628a790528072"
Nov 02 23:37:29.140888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:29.140513    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"
Nov 02 23:37:29.298410 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:29.298376    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"]
Nov 02 23:37:29.345776 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:29.345748    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5ff336-ff9d9875c-dggz2"]
Nov 02 23:37:29.747967 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:29.747905    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:29.923241 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:29.923202    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:30.093159 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:30.093084    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:30.548067 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:30.548040    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b91d5fc6-8e5d-44e9-be6d-58822e9697bb" path="/var/lib/kubelet/pods/b91d5fc6-8e5d-44e9-be6d-58822e9697bb/volumes"
Nov 02 23:37:32.455716 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:32.455682    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:32.616732 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:32.616703    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:34.021217 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:34.021185    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:34.481910 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:34.481830    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:35.200451 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:35.200400    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:36.258005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.257175    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"]
Nov 02 23:37:36.258544 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.258039    4294 cpu_manager.go:414] "containerMap: removing container" podUID="b91d5fc6-8e5d-44e9-be6d-58822e9697bb" containerName="maas-api"
Nov 02 23:37:36.258544 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.258060    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b91d5fc6-8e5d-44e9-be6d-58822e9697bb" containerName="maas-api"
Nov 02 23:37:36.258544 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.258252    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="b91d5fc6-8e5d-44e9-be6d-58822e9697bb" containerName="maas-api"
Nov 02 23:37:36.266266 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.266245    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.317098 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.317052    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-fb6e1d-openshift-default-dockercfg-5lz5h\""
Nov 02 23:37:36.335705 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.335672    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.335962 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.335930    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.336209 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.336192    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jgcz\" (UniqueName: \"kubernetes.io/projected/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-kube-api-access-7jgcz\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.336414 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.336399    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.336618 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.336603    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.336738 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.336725    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.336873 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.336860    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.337088 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.337022    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.337259 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.337073    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.380124 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.380040    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"]
Nov 02 23:37:36.444027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.443272    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.444027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.443384    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.444027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.443416    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.444027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.443463    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.444027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.443499    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.444027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.443548    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.444027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.443603    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.444027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.443651    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.444027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.443693    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jgcz\" (UniqueName: \"kubernetes.io/projected/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-kube-api-access-7jgcz\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.445772 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.445638    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.446552 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.446163    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.471286 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.451595    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.471286 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.451843    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.471286 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.452323    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.523656 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.489891    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.523656 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.523530    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.543536 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.543463    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.545511 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.545482    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jgcz\" (UniqueName: \"kubernetes.io/projected/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-kube-api-access-7jgcz\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.612034 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.587904    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:36.788732 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:36.787071    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:37.102951 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:37.102921    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"]
Nov 02 23:37:37.103754 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:37:37.103729    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbd38f7_a556_4fdf_acfc_9077aac74ec7.slice/crio-cb9fa0a2c94e51875df0d37b0f506f9cb9cac908a3fb0e4637736c662f5de710 WatchSource:0}: Error finding container cb9fa0a2c94e51875df0d37b0f506f9cb9cac908a3fb0e4637736c662f5de710: Status 404 returned error can't find the container with id cb9fa0a2c94e51875df0d37b0f506f9cb9cac908a3fb0e4637736c662f5de710
Nov 02 23:37:37.106249 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:37.106219    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:37:37.106317 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:37.106282    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:37:37.106354 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:37.106314    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:37:37.200227 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:37.200183    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5" event={"ID":"fdbd38f7-a556-4fdf-acfc-9077aac74ec7","Type":"ContainerStarted","Data":"602bd9a14dc375db51bcdca7e2894637533b92e7ea7c6f2da44b005cf4b091cc"}
Nov 02 23:37:37.200396 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:37.200298    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5" event={"ID":"fdbd38f7-a556-4fdf-acfc-9077aac74ec7","Type":"ContainerStarted","Data":"cb9fa0a2c94e51875df0d37b0f506f9cb9cac908a3fb0e4637736c662f5de710"}
Nov 02 23:37:37.314177 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:37.314122    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5" podStartSLOduration=1.314107537 podStartE2EDuration="1.314107537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:37:37.311420156 +0000 UTC m=+2696.879774942" watchObservedRunningTime="2026-11-02 23:37:37.314107537 +0000 UTC m=+2696.882462384"
Nov 02 23:37:37.588576 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:37.588544    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:38.590379 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:38.590344    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 23:37:38.590804 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:38.590401    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5" podUID="fdbd38f7-a556-4fdf-acfc-9077aac74ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 23:37:39.324541 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:39.324509    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:39.588802 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:39.588714    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 23:37:39.588802 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:39.588785    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5" podUID="fdbd38f7-a556-4fdf-acfc-9077aac74ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 23:37:39.881580 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:39.881498    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:40.223831 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:40.223733    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:40.390258 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:40.390226    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:40.589206 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:40.589174    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 23:37:40.589438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:40.589221    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5" podUID="fdbd38f7-a556-4fdf-acfc-9077aac74ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 23:37:41.589393 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:41.589306    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 23:37:41.589393 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:41.589366    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5" podUID="fdbd38f7-a556-4fdf-acfc-9077aac74ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 23:37:41.777597 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:37:41.777560    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:37:42.589142 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:42.589109    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 23:37:42.589334 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:42.589161    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5" podUID="fdbd38f7-a556-4fdf-acfc-9077aac74ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 23:37:43.588798 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:43.588760    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 02 23:37:43.589346 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:43.588823    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5" podUID="fdbd38f7-a556-4fdf-acfc-9077aac74ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 02 23:37:44.583432 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:44.583397    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:44.750446 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:44.750412    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:44.803829 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:44.803797    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:44.804284 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:44.804251    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:44.805138 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:44.805117    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:37:44.925633 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:44.925530    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:45.532892 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:45.532857    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:47.752890 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:47.752856    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:47.917571 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:47.917539    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:48.136793 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:48.136714    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg"]
Nov 02 23:37:48.141948 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:48.141931    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg"
Nov 02 23:37:48.212468 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:48.212424    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhgsf\" (UniqueName: \"kubernetes.io/projected/6665ed83-8dd6-4762-ad16-0b80d2b731f4-kube-api-access-bhgsf\") pod \"payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg\" (UID: \"6665ed83-8dd6-4762-ad16-0b80d2b731f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg"
Nov 02 23:37:48.212756 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:48.212735    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6665ed83-8dd6-4762-ad16-0b80d2b731f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg\" (UID: \"6665ed83-8dd6-4762-ad16-0b80d2b731f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg"
Nov 02 23:37:48.214938 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:48.214903    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg"]
Nov 02 23:37:48.251945 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:48.251903    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c5148c\""
Nov 02 23:37:48.252190 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:48.251919    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-c5148c-dockercfg-tbjxp\""
Nov 02 23:37:48.313952 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:48.313911    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhgsf\" (UniqueName: \"kubernetes.io/projected/6665ed83-8dd6-4762-ad16-0b80d2b731f4-kube-api-access-bhgsf\") pod \"payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg\" (UID: \"6665ed83-8dd6-4762-ad16-0b80d2b731f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg"
Nov 02 23:37:48.314160 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:48.313967    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6665ed83-8dd6-4762-ad16-0b80d2b731f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg\" (UID: \"6665ed83-8dd6-4762-ad16-0b80d2b731f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg"
Nov 02 23:37:48.314663 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:48.314645    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6665ed83-8dd6-4762-ad16-0b80d2b731f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg\" (UID: \"6665ed83-8dd6-4762-ad16-0b80d2b731f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg"
Nov 02 23:37:48.368607 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:48.368559    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhgsf\" (UniqueName: \"kubernetes.io/projected/6665ed83-8dd6-4762-ad16-0b80d2b731f4-kube-api-access-bhgsf\") pod \"payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg\" (UID: \"6665ed83-8dd6-4762-ad16-0b80d2b731f4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg"
Nov 02 23:37:48.454761 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:48.454680    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg"
Nov 02 23:37:48.836178 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:48.836146    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg"]
Nov 02 23:37:48.836962 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:37:48.836934    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6665ed83_8dd6_4762_ad16_0b80d2b731f4.slice/crio-9a5e773c6d8bf6b2947cbedf8a476b32971734b385f52ef805ff2ac043d5c1b1 WatchSource:0}: Error finding container 9a5e773c6d8bf6b2947cbedf8a476b32971734b385f52ef805ff2ac043d5c1b1: Status 404 returned error can't find the container with id 9a5e773c6d8bf6b2947cbedf8a476b32971734b385f52ef805ff2ac043d5c1b1
Nov 02 23:37:49.271530 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:49.271490    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg" event={"ID":"6665ed83-8dd6-4762-ad16-0b80d2b731f4","Type":"ContainerStarted","Data":"f6c249e382e9710d5e33bf2cada1ecf8203db32fedc3253988e2172247575e0c"}
Nov 02 23:37:49.271721 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:49.271549    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg" event={"ID":"6665ed83-8dd6-4762-ad16-0b80d2b731f4","Type":"ContainerStarted","Data":"9a5e773c6d8bf6b2947cbedf8a476b32971734b385f52ef805ff2ac043d5c1b1"}
Nov 02 23:37:49.271721 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:49.271584    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg"
Nov 02 23:37:49.381145 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:49.381093    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg" podStartSLOduration=1.3810771929999999 podStartE2EDuration="1.381077193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:37:49.379706458 +0000 UTC m=+2708.948061245" watchObservedRunningTime="2026-11-02 23:37:49.381077193 +0000 UTC m=+2708.949431978"
Nov 02 23:37:50.934528 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:50.934492    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:53.010332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:53.010301    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:53.714202 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:53.714168    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:54.106594 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:54.106537    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:55.676166 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:55.676129    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:56.087635 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:56.087599    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:56.254489 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:56.254456    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:56.418401 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:56.418317    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:57.712357 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:57.712326    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:57.895553 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:57.895501    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:37:58.073368 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:37:58.073279    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:00.295032 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:00.294132    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg"
Nov 02 23:38:00.996970 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:00.996894    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:01.176212 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:01.176173    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:01.480312 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:01.480232    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:01.529917 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:01.529891    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26"]
Nov 02 23:38:01.533490 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:01.533471    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26"
Nov 02 23:38:01.587194 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:01.587171    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-fe63ac\""
Nov 02 23:38:01.590118 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:01.590097    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26"]
Nov 02 23:38:01.608918 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:01.608484    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d025cd4-46da-4b22-af00-991525593a47-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26\" (UID: \"7d025cd4-46da-4b22-af00-991525593a47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26"
Nov 02 23:38:01.610851 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:01.610824    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz5wj\" (UniqueName: \"kubernetes.io/projected/7d025cd4-46da-4b22-af00-991525593a47-kube-api-access-vz5wj\") pod \"payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26\" (UID: \"7d025cd4-46da-4b22-af00-991525593a47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26"
Nov 02 23:38:01.653084 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:01.650316    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-fe63ac-dockercfg-88dps\""
Nov 02 23:38:01.668884 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:01.668793    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:01.712681 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:01.712646    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d025cd4-46da-4b22-af00-991525593a47-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26\" (UID: \"7d025cd4-46da-4b22-af00-991525593a47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26"
Nov 02 23:38:01.712876 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:01.712802    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz5wj\" (UniqueName: \"kubernetes.io/projected/7d025cd4-46da-4b22-af00-991525593a47-kube-api-access-vz5wj\") pod \"payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26\" (UID: \"7d025cd4-46da-4b22-af00-991525593a47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26"
Nov 02 23:38:01.713363 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:01.713345    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d025cd4-46da-4b22-af00-991525593a47-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26\" (UID: \"7d025cd4-46da-4b22-af00-991525593a47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26"
Nov 02 23:38:01.768749 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:01.768726    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz5wj\" (UniqueName: \"kubernetes.io/projected/7d025cd4-46da-4b22-af00-991525593a47-kube-api-access-vz5wj\") pod \"payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26\" (UID: \"7d025cd4-46da-4b22-af00-991525593a47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26"
Nov 02 23:38:01.846538 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:01.846434    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26"
Nov 02 23:38:01.861614 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:01.861587    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:02.038378 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:02.038271    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:02.218260 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:38:02.218226    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d025cd4_46da_4b22_af00_991525593a47.slice/crio-a89ca4b6ff0a46c70f59de8a108bab35f46716bb9761c43a541e091ffb354f22 WatchSource:0}: Error finding container a89ca4b6ff0a46c70f59de8a108bab35f46716bb9761c43a541e091ffb354f22: Status 404 returned error can't find the container with id a89ca4b6ff0a46c70f59de8a108bab35f46716bb9761c43a541e091ffb354f22
Nov 02 23:38:02.218493 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:02.218468    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26"]
Nov 02 23:38:02.363229 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:02.363143    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26" event={"ID":"7d025cd4-46da-4b22-af00-991525593a47","Type":"ContainerStarted","Data":"04474419fab214ee46463776e5eb4bed0ed2539cf7c80c2a9fad170af167b580"}
Nov 02 23:38:02.363229 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:02.363194    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26" event={"ID":"7d025cd4-46da-4b22-af00-991525593a47","Type":"ContainerStarted","Data":"a89ca4b6ff0a46c70f59de8a108bab35f46716bb9761c43a541e091ffb354f22"}
Nov 02 23:38:02.363229 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:02.363227    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26"
Nov 02 23:38:02.473607 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:02.473557    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26" podStartSLOduration=1.473543635 podStartE2EDuration="1.473543635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:02.471339021 +0000 UTC m=+2722.039693806" watchObservedRunningTime="2026-11-02 23:38:02.473543635 +0000 UTC m=+2722.041898439"
Nov 02 23:38:03.830213 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:03.830177    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:03.994649 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:03.994614    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:06.392535 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:06.392493    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:06.711064 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:06.710962    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:06.892073 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:06.892038    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:07.853429 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:07.853387    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:08.020160 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.020099    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:08.237676 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.237591    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt"]
Nov 02 23:38:08.241672 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.241656    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt"
Nov 02 23:38:08.292013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.291954    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-b4d53e\""
Nov 02 23:38:08.294736 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.294705    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4079e7c5-252d-40d6-ac37-67dde32884f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt\" (UID: \"4079e7c5-252d-40d6-ac37-67dde32884f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt"
Nov 02 23:38:08.294855 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.294835    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjm4x\" (UniqueName: \"kubernetes.io/projected/4079e7c5-252d-40d6-ac37-67dde32884f6-kube-api-access-zjm4x\") pod \"payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt\" (UID: \"4079e7c5-252d-40d6-ac37-67dde32884f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt"
Nov 02 23:38:08.355699 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.355667    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt"]
Nov 02 23:38:08.387966 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.387936    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-b4d53e-dockercfg-vl55w\""
Nov 02 23:38:08.396393 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.396360    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjm4x\" (UniqueName: \"kubernetes.io/projected/4079e7c5-252d-40d6-ac37-67dde32884f6-kube-api-access-zjm4x\") pod \"payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt\" (UID: \"4079e7c5-252d-40d6-ac37-67dde32884f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt"
Nov 02 23:38:08.396594 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.396578    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4079e7c5-252d-40d6-ac37-67dde32884f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt\" (UID: \"4079e7c5-252d-40d6-ac37-67dde32884f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt"
Nov 02 23:38:08.397314 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.397294    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4079e7c5-252d-40d6-ac37-67dde32884f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt\" (UID: \"4079e7c5-252d-40d6-ac37-67dde32884f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt"
Nov 02 23:38:08.404889 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.404863    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8"]
Nov 02 23:38:08.409770 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.409753    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8"
Nov 02 23:38:08.450820 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.450779    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjm4x\" (UniqueName: \"kubernetes.io/projected/4079e7c5-252d-40d6-ac37-67dde32884f6-kube-api-access-zjm4x\") pod \"payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt\" (UID: \"4079e7c5-252d-40d6-ac37-67dde32884f6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt"
Nov 02 23:38:08.462496 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.462457    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8"]
Nov 02 23:38:08.497684 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.497651    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/723b2ff2-4870-4de1-b820-6813a12699b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8\" (UID: \"723b2ff2-4870-4de1-b820-6813a12699b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8"
Nov 02 23:38:08.497909 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.497689    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swzx7\" (UniqueName: \"kubernetes.io/projected/723b2ff2-4870-4de1-b820-6813a12699b7-kube-api-access-swzx7\") pod \"payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8\" (UID: \"723b2ff2-4870-4de1-b820-6813a12699b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8"
Nov 02 23:38:08.558118 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.558081    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt"
Nov 02 23:38:08.600074 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.599558    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/723b2ff2-4870-4de1-b820-6813a12699b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8\" (UID: \"723b2ff2-4870-4de1-b820-6813a12699b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8"
Nov 02 23:38:08.600074 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.599620    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swzx7\" (UniqueName: \"kubernetes.io/projected/723b2ff2-4870-4de1-b820-6813a12699b7-kube-api-access-swzx7\") pod \"payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8\" (UID: \"723b2ff2-4870-4de1-b820-6813a12699b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8"
Nov 02 23:38:08.600294 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.600270    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/723b2ff2-4870-4de1-b820-6813a12699b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8\" (UID: \"723b2ff2-4870-4de1-b820-6813a12699b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8"
Nov 02 23:38:08.656173 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.656142    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swzx7\" (UniqueName: \"kubernetes.io/projected/723b2ff2-4870-4de1-b820-6813a12699b7-kube-api-access-swzx7\") pod \"payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8\" (UID: \"723b2ff2-4870-4de1-b820-6813a12699b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8"
Nov 02 23:38:08.727077 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.727044    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8"
Nov 02 23:38:08.936259 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:08.936234    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt"]
Nov 02 23:38:08.937508 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:38:08.937481    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4079e7c5_252d_40d6_ac37_67dde32884f6.slice/crio-1710a70fbb6f606313128ea41831026074160f4cf91b92a145bdaba849a36399 WatchSource:0}: Error finding container 1710a70fbb6f606313128ea41831026074160f4cf91b92a145bdaba849a36399: Status 404 returned error can't find the container with id 1710a70fbb6f606313128ea41831026074160f4cf91b92a145bdaba849a36399
Nov 02 23:38:09.103242 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:09.103218    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8"]
Nov 02 23:38:09.104382 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:38:09.104354    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod723b2ff2_4870_4de1_b820_6813a12699b7.slice/crio-6336b0f88a7e1258af52879472b9d2c6a19ea88d43c2f8ee029735bc81c0011d WatchSource:0}: Error finding container 6336b0f88a7e1258af52879472b9d2c6a19ea88d43c2f8ee029735bc81c0011d: Status 404 returned error can't find the container with id 6336b0f88a7e1258af52879472b9d2c6a19ea88d43c2f8ee029735bc81c0011d
Nov 02 23:38:09.412779 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:09.412693    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8" event={"ID":"723b2ff2-4870-4de1-b820-6813a12699b7","Type":"ContainerStarted","Data":"641736d12eaeee6ad944b7500388811a1bb711c7118cc95f122d7828641cbbeb"}
Nov 02 23:38:09.412779 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:09.412748    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8" event={"ID":"723b2ff2-4870-4de1-b820-6813a12699b7","Type":"ContainerStarted","Data":"6336b0f88a7e1258af52879472b9d2c6a19ea88d43c2f8ee029735bc81c0011d"}
Nov 02 23:38:09.412779 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:09.412778    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8"
Nov 02 23:38:09.414431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:09.414407    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt" event={"ID":"4079e7c5-252d-40d6-ac37-67dde32884f6","Type":"ContainerStarted","Data":"9b665ad8870f945e75a7c07c45d77ed5d2fb569cc5dc993a263084cfece37027"}
Nov 02 23:38:09.414539 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:09.414445    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt" event={"ID":"4079e7c5-252d-40d6-ac37-67dde32884f6","Type":"ContainerStarted","Data":"1710a70fbb6f606313128ea41831026074160f4cf91b92a145bdaba849a36399"}
Nov 02 23:38:09.414539 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:09.414495    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt"
Nov 02 23:38:09.523144 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:09.523094    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8" podStartSLOduration=1.52308087 podStartE2EDuration="1.52308087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:09.520339561 +0000 UTC m=+2729.088694349" watchObservedRunningTime="2026-11-02 23:38:09.52308087 +0000 UTC m=+2729.091435656"
Nov 02 23:38:09.629618 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:09.629560    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt" podStartSLOduration=1.6295451760000002 podStartE2EDuration="1.629545176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:09.628402355 +0000 UTC m=+2729.196757143" watchObservedRunningTime="2026-11-02 23:38:09.629545176 +0000 UTC m=+2729.197899963"
Nov 02 23:38:10.205835 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.205792    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:10.361147 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.361119    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:10.580638 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.580605    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76"]
Nov 02 23:38:10.585022 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.584973    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76"
Nov 02 23:38:10.636199 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.636166    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76"]
Nov 02 23:38:10.707399 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.707368    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-28f82c\""
Nov 02 23:38:10.707575 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.707511    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-28f82c-dockercfg-tmlkl\""
Nov 02 23:38:10.728245 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.728211    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95ad5c99-119f-46ca-8664-8f2878dbca94-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76\" (UID: \"95ad5c99-119f-46ca-8664-8f2878dbca94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76"
Nov 02 23:38:10.728411 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.728276    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcxqj\" (UniqueName: \"kubernetes.io/projected/95ad5c99-119f-46ca-8664-8f2878dbca94-kube-api-access-dcxqj\") pod \"payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76\" (UID: \"95ad5c99-119f-46ca-8664-8f2878dbca94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76"
Nov 02 23:38:10.756776 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.756745    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq"]
Nov 02 23:38:10.761075 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.761053    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq"
Nov 02 23:38:10.798891 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.798862    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq"]
Nov 02 23:38:10.830453 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.830230    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95ad5c99-119f-46ca-8664-8f2878dbca94-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76\" (UID: \"95ad5c99-119f-46ca-8664-8f2878dbca94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76"
Nov 02 23:38:10.830735 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.830679    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcxqj\" (UniqueName: \"kubernetes.io/projected/95ad5c99-119f-46ca-8664-8f2878dbca94-kube-api-access-dcxqj\") pod \"payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76\" (UID: \"95ad5c99-119f-46ca-8664-8f2878dbca94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76"
Nov 02 23:38:10.830832 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.830783    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnvn2\" (UniqueName: \"kubernetes.io/projected/36cb1d31-8467-441f-90bf-07ba20b0f3c6-kube-api-access-hnvn2\") pod \"payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq\" (UID: \"36cb1d31-8467-441f-90bf-07ba20b0f3c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq"
Nov 02 23:38:10.830893 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.830843    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36cb1d31-8467-441f-90bf-07ba20b0f3c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq\" (UID: \"36cb1d31-8467-441f-90bf-07ba20b0f3c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq"
Nov 02 23:38:10.831567 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.831536    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95ad5c99-119f-46ca-8664-8f2878dbca94-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76\" (UID: \"95ad5c99-119f-46ca-8664-8f2878dbca94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76"
Nov 02 23:38:10.886217 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.886177    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcxqj\" (UniqueName: \"kubernetes.io/projected/95ad5c99-119f-46ca-8664-8f2878dbca94-kube-api-access-dcxqj\") pod \"payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76\" (UID: \"95ad5c99-119f-46ca-8664-8f2878dbca94\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76"
Nov 02 23:38:10.899557 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.899530    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76"
Nov 02 23:38:10.931959 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.931923    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnvn2\" (UniqueName: \"kubernetes.io/projected/36cb1d31-8467-441f-90bf-07ba20b0f3c6-kube-api-access-hnvn2\") pod \"payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq\" (UID: \"36cb1d31-8467-441f-90bf-07ba20b0f3c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq"
Nov 02 23:38:10.932127 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.931971    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36cb1d31-8467-441f-90bf-07ba20b0f3c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq\" (UID: \"36cb1d31-8467-441f-90bf-07ba20b0f3c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq"
Nov 02 23:38:10.932689 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.932665    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36cb1d31-8467-441f-90bf-07ba20b0f3c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq\" (UID: \"36cb1d31-8467-441f-90bf-07ba20b0f3c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq"
Nov 02 23:38:10.990500 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:10.990471    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnvn2\" (UniqueName: \"kubernetes.io/projected/36cb1d31-8467-441f-90bf-07ba20b0f3c6-kube-api-access-hnvn2\") pod \"payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq\" (UID: \"36cb1d31-8467-441f-90bf-07ba20b0f3c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq"
Nov 02 23:38:11.076103 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:11.076069    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq"
Nov 02 23:38:11.267012 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:11.266969    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76"]
Nov 02 23:38:11.268428 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:38:11.268398    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ad5c99_119f_46ca_8664_8f2878dbca94.slice/crio-c0f1deb273b9fe1c7f964635cea9af49513ec50621d1b731a3ee0eac3c880be4 WatchSource:0}: Error finding container c0f1deb273b9fe1c7f964635cea9af49513ec50621d1b731a3ee0eac3c880be4: Status 404 returned error can't find the container with id c0f1deb273b9fe1c7f964635cea9af49513ec50621d1b731a3ee0eac3c880be4
Nov 02 23:38:11.429696 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:11.429592    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76" event={"ID":"95ad5c99-119f-46ca-8664-8f2878dbca94","Type":"ContainerStarted","Data":"f18166f5e45be864f02b4bf751c71292a3084367dcb9f04b632d389284f77dea"}
Nov 02 23:38:11.429696 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:11.429654    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76" event={"ID":"95ad5c99-119f-46ca-8664-8f2878dbca94","Type":"ContainerStarted","Data":"c0f1deb273b9fe1c7f964635cea9af49513ec50621d1b731a3ee0eac3c880be4"}
Nov 02 23:38:11.429696 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:11.429685    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76"
Nov 02 23:38:11.451882 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:11.451845    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq"]
Nov 02 23:38:11.455045 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:38:11.455021    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36cb1d31_8467_441f_90bf_07ba20b0f3c6.slice/crio-f3a196d7f8458d5d331253459563c454ce5d03f5705dfb888f0dae9bae065699 WatchSource:0}: Error finding container f3a196d7f8458d5d331253459563c454ce5d03f5705dfb888f0dae9bae065699: Status 404 returned error can't find the container with id f3a196d7f8458d5d331253459563c454ce5d03f5705dfb888f0dae9bae065699
Nov 02 23:38:11.538792 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:11.538744    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76" podStartSLOduration=1.538730422 podStartE2EDuration="1.538730422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:11.53737242 +0000 UTC m=+2731.105727207" watchObservedRunningTime="2026-11-02 23:38:11.538730422 +0000 UTC m=+2731.107085224"
Nov 02 23:38:11.755324 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:38:11.755287    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:38:12.193700 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:12.193613    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:12.437744 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:12.437705    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq" event={"ID":"36cb1d31-8467-441f-90bf-07ba20b0f3c6","Type":"ContainerStarted","Data":"a2f19b1757ca0654957594e93f52980ad5d3a0302b8034a247d47c984c66ac93"}
Nov 02 23:38:12.438254 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:12.437767    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq"
Nov 02 23:38:12.438254 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:12.437784    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq" event={"ID":"36cb1d31-8467-441f-90bf-07ba20b0f3c6","Type":"ContainerStarted","Data":"f3a196d7f8458d5d331253459563c454ce5d03f5705dfb888f0dae9bae065699"}
Nov 02 23:38:12.548554 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:12.548497    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq" podStartSLOduration=2.548477515 podStartE2EDuration="2.548477515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:12.546113591 +0000 UTC m=+2732.114468379" watchObservedRunningTime="2026-11-02 23:38:12.548477515 +0000 UTC m=+2732.116832303"
Nov 02 23:38:13.376082 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:13.376051    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26"
Nov 02 23:38:15.115380 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:15.115338    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:16.258882 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:16.258819    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:16.886563 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:16.886527    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:17.150400 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:17.150303    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:17.317621 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:17.317590    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:17.485702 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:17.485609    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:20.423622 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:20.423594    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8"
Nov 02 23:38:20.424048 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:20.423667    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt"
Nov 02 23:38:21.149809 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:21.149762    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:21.323705 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:21.323658    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:22.441694 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:22.441660    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76"
Nov 02 23:38:22.485946 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:22.485912    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"]
Nov 02 23:38:22.490184 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:22.490169    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"
Nov 02 23:38:22.540347 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:22.540322    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-c5148c-serving-cert\""
Nov 02 23:38:22.549714 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:22.549689    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"]
Nov 02 23:38:22.577039 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:22.577014    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqvqg\" (UniqueName: \"kubernetes.io/projected/4d799ed5-6504-4440-aa0f-84ba25128c25-kube-api-access-rqvqg\") pod \"maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj\" (UID: \"4d799ed5-6504-4440-aa0f-84ba25128c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"
Nov 02 23:38:22.577208 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:22.577149    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d799ed5-6504-4440-aa0f-84ba25128c25-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj\" (UID: \"4d799ed5-6504-4440-aa0f-84ba25128c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"
Nov 02 23:38:22.577282 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:22.577266    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d799ed5-6504-4440-aa0f-84ba25128c25-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj\" (UID: \"4d799ed5-6504-4440-aa0f-84ba25128c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"
Nov 02 23:38:22.577415 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:22.577394    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d799ed5-6504-4440-aa0f-84ba25128c25-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj\" (UID: \"4d799ed5-6504-4440-aa0f-84ba25128c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"
Nov 02 23:38:22.678429 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:22.678396    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d799ed5-6504-4440-aa0f-84ba25128c25-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj\" (UID: \"4d799ed5-6504-4440-aa0f-84ba25128c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"
Nov 02 23:38:22.678630 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:22.678485    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d799ed5-6504-4440-aa0f-84ba25128c25-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj\" (UID: \"4d799ed5-6504-4440-aa0f-84ba25128c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"
Nov 02 23:38:22.678630 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:22.678554    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d799ed5-6504-4440-aa0f-84ba25128c25-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj\" (UID: \"4d799ed5-6504-4440-aa0f-84ba25128c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"
Nov 02 23:38:22.678630 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:22.678606    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqvqg\" (UniqueName: \"kubernetes.io/projected/4d799ed5-6504-4440-aa0f-84ba25128c25-kube-api-access-rqvqg\") pod \"maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj\" (UID: \"4d799ed5-6504-4440-aa0f-84ba25128c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"
Nov 02 23:38:22.678776 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:38:22.678707    4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-serving-cert: secret "maas-api-e2e-shared-a-w2-c5148c-serving-cert" not found
Nov 02 23:38:22.678818 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:38:22.678774    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d799ed5-6504-4440-aa0f-84ba25128c25-maas-api-tls podName:4d799ed5-6504-4440-aa0f-84ba25128c25 nodeName:}" failed. No retries permitted until 2026-11-02 23:38:23.178754527 +0000 UTC m=+2742.747109292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d799ed5-6504-4440-aa0f-84ba25128c25-maas-api-tls") pod "maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj" (UID: "4d799ed5-6504-4440-aa0f-84ba25128c25") : secret "maas-api-e2e-shared-a-w2-c5148c-serving-cert" not found
Nov 02 23:38:22.679127 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:22.679105    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d799ed5-6504-4440-aa0f-84ba25128c25-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj\" (UID: \"4d799ed5-6504-4440-aa0f-84ba25128c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"
Nov 02 23:38:22.680880 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:22.680859    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d799ed5-6504-4440-aa0f-84ba25128c25-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj\" (UID: \"4d799ed5-6504-4440-aa0f-84ba25128c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"
Nov 02 23:38:22.704435 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:22.704348    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:22.733401 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:22.733366    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqvqg\" (UniqueName: \"kubernetes.io/projected/4d799ed5-6504-4440-aa0f-84ba25128c25-kube-api-access-rqvqg\") pod \"maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj\" (UID: \"4d799ed5-6504-4440-aa0f-84ba25128c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"
Nov 02 23:38:22.861869 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:22.861832    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:23.029654 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:23.029617    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:23.185509 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:23.185476    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d799ed5-6504-4440-aa0f-84ba25128c25-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj\" (UID: \"4d799ed5-6504-4440-aa0f-84ba25128c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"
Nov 02 23:38:23.187860 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:23.187836    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d799ed5-6504-4440-aa0f-84ba25128c25-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj\" (UID: \"4d799ed5-6504-4440-aa0f-84ba25128c25\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"
Nov 02 23:38:23.405172 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:23.405083    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"
Nov 02 23:38:23.448219 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:23.448187    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq"
Nov 02 23:38:23.773785 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:23.773753    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"]
Nov 02 23:38:23.774249 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:38:23.774213    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d799ed5_6504_4440_aa0f_84ba25128c25.slice/crio-09d96c871ff01d4182b09b4fe2f70f55e2bc5899c6ebc1787dc402852d21f1c4 WatchSource:0}: Error finding container 09d96c871ff01d4182b09b4fe2f70f55e2bc5899c6ebc1787dc402852d21f1c4: Status 404 returned error can't find the container with id 09d96c871ff01d4182b09b4fe2f70f55e2bc5899c6ebc1787dc402852d21f1c4
Nov 02 23:38:24.518567 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:24.518531    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj" event={"ID":"4d799ed5-6504-4440-aa0f-84ba25128c25","Type":"ContainerStarted","Data":"09d96c871ff01d4182b09b4fe2f70f55e2bc5899c6ebc1787dc402852d21f1c4"}
Nov 02 23:38:25.526595 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:25.526560    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj" event={"ID":"4d799ed5-6504-4440-aa0f-84ba25128c25","Type":"ContainerStarted","Data":"a4e1499cf29d501f21a352d6ff10f424df259be8373896e42398e705f2840fd9"}
Nov 02 23:38:25.527033 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:25.526634    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"
Nov 02 23:38:25.637674 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:25.637622    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj" podStartSLOduration=2.753654443 podStartE2EDuration="3.637608381s" totalImagesPullingTime="883.953938ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:25.636690936 +0000 UTC m=+2745.205045723" watchObservedRunningTime="2026-11-02 23:38:25.637608381 +0000 UTC m=+2745.205963168"
Nov 02 23:38:25.670707 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:25.670668    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:25.873685 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:25.873590    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:26.207890 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:26.204390    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph"]
Nov 02 23:38:26.209781 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:26.209757    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph"
Nov 02 23:38:26.254140 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:26.254108    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph"]
Nov 02 23:38:26.260364 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:26.260335    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-b1be95\""
Nov 02 23:38:26.297141 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:26.297111    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-b1be95-dockercfg-qf9v9\""
Nov 02 23:38:26.324242 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:26.324213    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b96272-5bca-4475-84bb-6022ec383f6f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph\" (UID: \"d2b96272-5bca-4475-84bb-6022ec383f6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph"
Nov 02 23:38:26.324242 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:26.324246    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69kwp\" (UniqueName: \"kubernetes.io/projected/d2b96272-5bca-4475-84bb-6022ec383f6f-kube-api-access-69kwp\") pod \"payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph\" (UID: \"d2b96272-5bca-4475-84bb-6022ec383f6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph"
Nov 02 23:38:26.425467 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:26.425432    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b96272-5bca-4475-84bb-6022ec383f6f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph\" (UID: \"d2b96272-5bca-4475-84bb-6022ec383f6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph"
Nov 02 23:38:26.425467 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:26.425472    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69kwp\" (UniqueName: \"kubernetes.io/projected/d2b96272-5bca-4475-84bb-6022ec383f6f-kube-api-access-69kwp\") pod \"payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph\" (UID: \"d2b96272-5bca-4475-84bb-6022ec383f6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph"
Nov 02 23:38:26.426247 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:26.426226    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b96272-5bca-4475-84bb-6022ec383f6f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph\" (UID: \"d2b96272-5bca-4475-84bb-6022ec383f6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph"
Nov 02 23:38:26.480263 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:26.480190    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69kwp\" (UniqueName: \"kubernetes.io/projected/d2b96272-5bca-4475-84bb-6022ec383f6f-kube-api-access-69kwp\") pod \"payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph\" (UID: \"d2b96272-5bca-4475-84bb-6022ec383f6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph"
Nov 02 23:38:26.524394 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:26.524345    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph"
Nov 02 23:38:26.897337 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:26.897301    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph"]
Nov 02 23:38:26.897781 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:38:26.897509    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b96272_5bca_4475_84bb_6022ec383f6f.slice/crio-df3ef12dcada36d470f2ee10ecb8e43a045a381b62b65d60163097db80b0afff WatchSource:0}: Error finding container df3ef12dcada36d470f2ee10ecb8e43a045a381b62b65d60163097db80b0afff: Status 404 returned error can't find the container with id df3ef12dcada36d470f2ee10ecb8e43a045a381b62b65d60163097db80b0afff
Nov 02 23:38:27.539901 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:27.539852    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph" event={"ID":"d2b96272-5bca-4475-84bb-6022ec383f6f","Type":"ContainerStarted","Data":"83146dbf02ece98d6fb15e076922b976a287ed823f0734be809f3a1e6fdd7c0a"}
Nov 02 23:38:27.540145 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:27.539920    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph" event={"ID":"d2b96272-5bca-4475-84bb-6022ec383f6f","Type":"ContainerStarted","Data":"df3ef12dcada36d470f2ee10ecb8e43a045a381b62b65d60163097db80b0afff"}
Nov 02 23:38:27.540145 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:27.540011    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph"
Nov 02 23:38:27.651284 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:27.651231    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph" podStartSLOduration=1.651218841 podStartE2EDuration="1.651218841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:27.648503136 +0000 UTC m=+2747.216857924" watchObservedRunningTime="2026-11-02 23:38:27.651218841 +0000 UTC m=+2747.219573647"
Nov 02 23:38:28.416697 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:28.416668    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:31.538122 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:31.538089    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"
Nov 02 23:38:33.278193 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:33.278154    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:37.143204 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:37.143162    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:37.877850 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:37.877817    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"]
Nov 02 23:38:37.882380 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:37.882351    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"
Nov 02 23:38:37.935380 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:37.935340    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"]
Nov 02 23:38:37.969542 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:37.969508    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/484b42ed-574b-4592-887f-1e85faa347ac-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7\" (UID: \"484b42ed-574b-4592-887f-1e85faa347ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"
Nov 02 23:38:37.969703 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:37.969685    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/484b42ed-574b-4592-887f-1e85faa347ac-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7\" (UID: \"484b42ed-574b-4592-887f-1e85faa347ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"
Nov 02 23:38:37.969758 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:37.969724    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/484b42ed-574b-4592-887f-1e85faa347ac-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7\" (UID: \"484b42ed-574b-4592-887f-1e85faa347ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"
Nov 02 23:38:37.969821 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:37.969805    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6crb\" (UniqueName: \"kubernetes.io/projected/484b42ed-574b-4592-887f-1e85faa347ac-kube-api-access-j6crb\") pod \"maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7\" (UID: \"484b42ed-574b-4592-887f-1e85faa347ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"
Nov 02 23:38:37.978199 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:37.978176    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-fe63ac-serving-cert\""
Nov 02 23:38:38.071390 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:38.071347    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6crb\" (UniqueName: \"kubernetes.io/projected/484b42ed-574b-4592-887f-1e85faa347ac-kube-api-access-j6crb\") pod \"maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7\" (UID: \"484b42ed-574b-4592-887f-1e85faa347ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"
Nov 02 23:38:38.071557 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:38.071444    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/484b42ed-574b-4592-887f-1e85faa347ac-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7\" (UID: \"484b42ed-574b-4592-887f-1e85faa347ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"
Nov 02 23:38:38.071557 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:38.071552    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/484b42ed-574b-4592-887f-1e85faa347ac-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7\" (UID: \"484b42ed-574b-4592-887f-1e85faa347ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"
Nov 02 23:38:38.071635 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:38.071582    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/484b42ed-574b-4592-887f-1e85faa347ac-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7\" (UID: \"484b42ed-574b-4592-887f-1e85faa347ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"
Nov 02 23:38:38.071715 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:38:38.071699    4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-serving-cert: secret "maas-api-e2e-shared-a-w1-fe63ac-serving-cert" not found
Nov 02 23:38:38.071793 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:38:38.071782    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/484b42ed-574b-4592-887f-1e85faa347ac-maas-api-tls podName:484b42ed-574b-4592-887f-1e85faa347ac nodeName:}" failed. No retries permitted until 2026-11-02 23:38:38.571760253 +0000 UTC m=+2758.140115034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/484b42ed-574b-4592-887f-1e85faa347ac-maas-api-tls") pod "maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7" (UID: "484b42ed-574b-4592-887f-1e85faa347ac") : secret "maas-api-e2e-shared-a-w1-fe63ac-serving-cert" not found
Nov 02 23:38:38.072296 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:38.072276    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/484b42ed-574b-4592-887f-1e85faa347ac-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7\" (UID: \"484b42ed-574b-4592-887f-1e85faa347ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"
Nov 02 23:38:38.073823 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:38.073801    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/484b42ed-574b-4592-887f-1e85faa347ac-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7\" (UID: \"484b42ed-574b-4592-887f-1e85faa347ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"
Nov 02 23:38:38.109657 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:38.109621    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:38.126453 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:38.126425    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6crb\" (UniqueName: \"kubernetes.io/projected/484b42ed-574b-4592-887f-1e85faa347ac-kube-api-access-j6crb\") pod \"maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7\" (UID: \"484b42ed-574b-4592-887f-1e85faa347ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"
Nov 02 23:38:38.273234 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:38.273196    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:38.549551 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:38.549469    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph"
Nov 02 23:38:38.576578 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:38.576543    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/484b42ed-574b-4592-887f-1e85faa347ac-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7\" (UID: \"484b42ed-574b-4592-887f-1e85faa347ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"
Nov 02 23:38:38.579233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:38.579206    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/484b42ed-574b-4592-887f-1e85faa347ac-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7\" (UID: \"484b42ed-574b-4592-887f-1e85faa347ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"
Nov 02 23:38:38.700325 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:38.700294    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:38.795992 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:38.795945    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"
Nov 02 23:38:39.214198 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:39.214170    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"]
Nov 02 23:38:39.214894 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:38:39.214865    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484b42ed_574b_4592_887f_1e85faa347ac.slice/crio-ecd62387e8295bf6ec08ed34bd0c7f8a996033f7bbfff0271bd4b838610e7589 WatchSource:0}: Error finding container ecd62387e8295bf6ec08ed34bd0c7f8a996033f7bbfff0271bd4b838610e7589: Status 404 returned error can't find the container with id ecd62387e8295bf6ec08ed34bd0c7f8a996033f7bbfff0271bd4b838610e7589
Nov 02 23:38:39.216488 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:39.216468    4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 23:38:39.620428 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:39.620394    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7" event={"ID":"484b42ed-574b-4592-887f-1e85faa347ac","Type":"ContainerStarted","Data":"ecd62387e8295bf6ec08ed34bd0c7f8a996033f7bbfff0271bd4b838610e7589"}
Nov 02 23:38:40.629108 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:40.629016    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7" event={"ID":"484b42ed-574b-4592-887f-1e85faa347ac","Type":"ContainerStarted","Data":"007010af3804f06023786c7b309a80ea1271a0464e8a9f8f37aa3b23bc361658"}
Nov 02 23:38:40.629564 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:40.629136    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"
Nov 02 23:38:40.753644 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:40.753588    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7" podStartSLOduration=2.722977986 podStartE2EDuration="3.753574408s" totalImagesPullingTime="1.030596422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:38:40.751002512 +0000 UTC m=+2760.319357293" watchObservedRunningTime="2026-11-02 23:38:40.753574408 +0000 UTC m=+2760.321929195"
Nov 02 23:38:41.750961 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:38:41.750922    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:38:43.170600 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:43.170565    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:46.641136 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:46.641105    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"
Nov 02 23:38:48.005063 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:48.005025    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:48.163695 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:48.163661    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:48.320093 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:48.320004    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:53.100326 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:53.100288    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:58.018925 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:58.018893    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:58.218216 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:58.218184    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:38:58.392102 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:38:58.392028    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:00.039489 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:00.039446    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:03.117286 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:03.117248    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:08.061050 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:08.061010    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:11.750773 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:39:11.750729    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:39:12.998652 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:12.998611    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:17.582180 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:17.582141    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"]
Nov 02 23:39:17.586518 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:17.586495    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"
Nov 02 23:39:17.639971 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:17.639944    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"]
Nov 02 23:39:17.683496 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:17.683470    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-b1be95-serving-cert\""
Nov 02 23:39:17.749282 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:17.749247    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80f7343-30ef-4b44-9e28-c3c7a9e80952-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds\" (UID: \"a80f7343-30ef-4b44-9e28-c3c7a9e80952\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"
Nov 02 23:39:17.749485 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:17.749291    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9q9f\" (UniqueName: \"kubernetes.io/projected/a80f7343-30ef-4b44-9e28-c3c7a9e80952-kube-api-access-m9q9f\") pod \"maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds\" (UID: \"a80f7343-30ef-4b44-9e28-c3c7a9e80952\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"
Nov 02 23:39:17.749485 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:17.749376    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a80f7343-30ef-4b44-9e28-c3c7a9e80952-metrics-certs\") pod \"maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds\" (UID: \"a80f7343-30ef-4b44-9e28-c3c7a9e80952\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"
Nov 02 23:39:17.749485 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:17.749425    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a80f7343-30ef-4b44-9e28-c3c7a9e80952-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds\" (UID: \"a80f7343-30ef-4b44-9e28-c3c7a9e80952\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"
Nov 02 23:39:17.818551 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:17.816470    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:17.850534 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:17.850444    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80f7343-30ef-4b44-9e28-c3c7a9e80952-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds\" (UID: \"a80f7343-30ef-4b44-9e28-c3c7a9e80952\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"
Nov 02 23:39:17.850534 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:17.850512    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9q9f\" (UniqueName: \"kubernetes.io/projected/a80f7343-30ef-4b44-9e28-c3c7a9e80952-kube-api-access-m9q9f\") pod \"maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds\" (UID: \"a80f7343-30ef-4b44-9e28-c3c7a9e80952\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"
Nov 02 23:39:17.850859 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:17.850578    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a80f7343-30ef-4b44-9e28-c3c7a9e80952-metrics-certs\") pod \"maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds\" (UID: \"a80f7343-30ef-4b44-9e28-c3c7a9e80952\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"
Nov 02 23:39:17.850859 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:17.850627    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a80f7343-30ef-4b44-9e28-c3c7a9e80952-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds\" (UID: \"a80f7343-30ef-4b44-9e28-c3c7a9e80952\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"
Nov 02 23:39:17.851707 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:17.851680    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80f7343-30ef-4b44-9e28-c3c7a9e80952-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds\" (UID: \"a80f7343-30ef-4b44-9e28-c3c7a9e80952\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"
Nov 02 23:39:17.854541 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:17.854494    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a80f7343-30ef-4b44-9e28-c3c7a9e80952-maas-api-tls\") pod \"maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds\" (UID: \"a80f7343-30ef-4b44-9e28-c3c7a9e80952\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"
Nov 02 23:39:17.861867 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:17.861813    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a80f7343-30ef-4b44-9e28-c3c7a9e80952-metrics-certs\") pod \"maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds\" (UID: \"a80f7343-30ef-4b44-9e28-c3c7a9e80952\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"
Nov 02 23:39:17.905709 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:17.905679    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9q9f\" (UniqueName: \"kubernetes.io/projected/a80f7343-30ef-4b44-9e28-c3c7a9e80952-kube-api-access-m9q9f\") pod \"maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds\" (UID: \"a80f7343-30ef-4b44-9e28-c3c7a9e80952\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"
Nov 02 23:39:17.986354 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:17.986295    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:18.158790 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:18.158703    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:18.200083 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:18.200054    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"
Nov 02 23:39:18.580811 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:18.580788    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"]
Nov 02 23:39:18.582349 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:39:18.582323    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80f7343_30ef_4b44_9e28_c3c7a9e80952.slice/crio-135a6f1d30dc00aa3b8b1df8307a23f89da7cd621357411319e3ad6d83fbad3e WatchSource:0}: Error finding container 135a6f1d30dc00aa3b8b1df8307a23f89da7cd621357411319e3ad6d83fbad3e: Status 404 returned error can't find the container with id 135a6f1d30dc00aa3b8b1df8307a23f89da7cd621357411319e3ad6d83fbad3e
Nov 02 23:39:18.868604 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:18.868525    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds" event={"ID":"a80f7343-30ef-4b44-9e28-c3c7a9e80952","Type":"ContainerStarted","Data":"135a6f1d30dc00aa3b8b1df8307a23f89da7cd621357411319e3ad6d83fbad3e"}
Nov 02 23:39:19.876752 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:19.876662    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds" event={"ID":"a80f7343-30ef-4b44-9e28-c3c7a9e80952","Type":"ContainerStarted","Data":"327d3030dac889df6ee35645de7ee64df10569e3c4f85b82e42ed532edffecc5"}
Nov 02 23:39:19.876752 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:19.876722    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"
Nov 02 23:39:19.988514 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:19.988461    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds" podStartSLOduration=2.034650833 podStartE2EDuration="2.988448842s" totalImagesPullingTime="953.798009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:39:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:39:19.985241958 +0000 UTC m=+2799.553596744" watchObservedRunningTime="2026-11-02 23:39:19.988448842 +0000 UTC m=+2799.556803627"
Nov 02 23:39:22.883078 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:22.883048    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:23.592641 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:23.592605    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:25.891489 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:25.891462    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"
Nov 02 23:39:27.774899 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:27.774870    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:31.552618 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:31.552587    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:31.959249 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:31.959149    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:32.555625 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:32.555584    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:32.992396 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:32.992361    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:33.233718 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:33.233681    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:36.595389 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:36.595336    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:36.762433 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:36.762401    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:36.929098 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:36.929022    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:37.582121 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:37.582080    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:38.680345 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:38.680313    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:39.746437 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:39.746406    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:39.906611 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:39.906579    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:41.306059 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:41.306030    4294 scope.go:122] "RemoveContainer" containerID="f9fad49ba7fc362b6f9529eb63d9d3c20f81b357ecfbca9efb8061f121d0d2e3"
Nov 02 23:39:41.751668 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:39:41.751629    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:39:43.815131 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:43.815099    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:44.679355 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:44.679313    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:45.129710 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:45.129669    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:46.389229 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.389191    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"]
Nov 02 23:39:46.394936 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.394912    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.465045 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.465016    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-c25940-openshift-default-dockercfg-h6tvp\""
Nov 02 23:39:46.496863 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.496835    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"]
Nov 02 23:39:46.503476 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.503449    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-token\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.503638 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.503514    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-credential-socket\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.503638 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.503566    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-envoy\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.503638 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.503616    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-data\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.503838 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.503669    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istiod-ca-cert\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.503838 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.503723    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-workload-socket\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.503838 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.503751    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-podinfo\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.503838 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.503781    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzg7v\" (UniqueName: \"kubernetes.io/projected/30dbc36b-dc80-47c8-a95a-5ad407f121a1-kube-api-access-fzg7v\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.504314 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.503855    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-workload-certs\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.529719 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.529690    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:46.605192 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.605150    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-workload-certs\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.605385 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.605306    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-token\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.605385 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.605364    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-credential-socket\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.605517 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.605398    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-envoy\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.605517 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.605432    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-data\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.605517 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.605475    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istiod-ca-cert\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.605689 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.605535    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-workload-socket\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.605689 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.605542    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-workload-certs\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.605689 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.605575    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-podinfo\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.605689 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.605614    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzg7v\" (UniqueName: \"kubernetes.io/projected/30dbc36b-dc80-47c8-a95a-5ad407f121a1-kube-api-access-fzg7v\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.605952 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.605706    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-credential-socket\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.605952 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.605845    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-data\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.606100 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.605969    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-workload-socket\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.606236 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.606217    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istiod-ca-cert\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.607846 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.607822    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-envoy\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.608111 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.608094    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-podinfo\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.660065 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.659963    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-token\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.660205 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.660058    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzg7v\" (UniqueName: \"kubernetes.io/projected/30dbc36b-dc80-47c8-a95a-5ad407f121a1-kube-api-access-fzg7v\") pod \"e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") " pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:46.699680 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.699643    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:46.708696 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:46.708670    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:47.136395 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:39:47.136369    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30dbc36b_dc80_47c8_a95a_5ad407f121a1.slice/crio-8075b9adedd649e7b9e8ec03a10545929a0cd819958c7f949636a1dae344dbe8 WatchSource:0}: Error finding container 8075b9adedd649e7b9e8ec03a10545929a0cd819958c7f949636a1dae344dbe8: Status 404 returned error can't find the container with id 8075b9adedd649e7b9e8ec03a10545929a0cd819958c7f949636a1dae344dbe8
Nov 02 23:39:47.137514 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:47.137487    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"]
Nov 02 23:39:47.139267 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:47.139236    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:39:47.139377 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:47.139298    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:39:47.139377 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:47.139330    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:39:48.083898 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:48.083863    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb" event={"ID":"30dbc36b-dc80-47c8-a95a-5ad407f121a1","Type":"ContainerStarted","Data":"97abd1623f7b267f2e098831e569610e8eef2a932a8321904ca8eabd028014b0"}
Nov 02 23:39:48.084322 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:48.083921    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb" event={"ID":"30dbc36b-dc80-47c8-a95a-5ad407f121a1","Type":"ContainerStarted","Data":"8075b9adedd649e7b9e8ec03a10545929a0cd819958c7f949636a1dae344dbe8"}
Nov 02 23:39:48.196751 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:48.196700    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb" podStartSLOduration=2.196685975 podStartE2EDuration="2.196685975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:39:48.194668378 +0000 UTC m=+2827.763023165" watchObservedRunningTime="2026-11-02 23:39:48.196685975 +0000 UTC m=+2827.765040814"
Nov 02 23:39:48.709060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:48.709018    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:48.714812 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:48.714788    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:49.090045 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:49.090014    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:49.090830 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:49.090810    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:39:49.114993 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:49.114954    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:51.029337 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:51.029302    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:51.203758 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:51.203725    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:51.383534 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:51.383452    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:54.168248 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:54.168172    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:54.339792 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:54.339753    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:54.500782 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:54.500752    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:57.932808 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:57.932774    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:58.354166 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:58.354125    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:58.947014 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:58.946958    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:59.447767 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:59.447733    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:39:59.732517 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:39:59.732435    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:03.156785 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:03.156755    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:03.335906 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:03.335873    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:03.506023 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:03.505958    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:05.190212 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:05.190182    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:07.981270 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:07.981235    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:08.390784 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:08.390753    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:09.098629 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:09.098594    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:09.520802 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:09.520764    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:10.436043 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:10.436013    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:11.755922 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:40:11.755888    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:40:12.456479 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:12.456445    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:12.644948 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:12.644901    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:12.729466 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:12.729382    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6"]
Nov 02 23:40:12.733408 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:12.733388    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6"
Nov 02 23:40:12.783812 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:12.783786    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-a51378\""
Nov 02 23:40:12.786572 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:12.786548    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6"]
Nov 02 23:40:12.851004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:12.844324    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-a51378-dockercfg-kllfp\""
Nov 02 23:40:12.851004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:12.847319    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ldwh\" (UniqueName: \"kubernetes.io/projected/fba0cc63-6a7a-474c-b6e4-c411f912eb46-kube-api-access-4ldwh\") pod \"payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6\" (UID: \"fba0cc63-6a7a-474c-b6e4-c411f912eb46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6"
Nov 02 23:40:12.851004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:12.847481    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba0cc63-6a7a-474c-b6e4-c411f912eb46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6\" (UID: \"fba0cc63-6a7a-474c-b6e4-c411f912eb46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6"
Nov 02 23:40:12.948922 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:12.948881    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ldwh\" (UniqueName: \"kubernetes.io/projected/fba0cc63-6a7a-474c-b6e4-c411f912eb46-kube-api-access-4ldwh\") pod \"payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6\" (UID: \"fba0cc63-6a7a-474c-b6e4-c411f912eb46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6"
Nov 02 23:40:12.949140 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:12.949025    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba0cc63-6a7a-474c-b6e4-c411f912eb46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6\" (UID: \"fba0cc63-6a7a-474c-b6e4-c411f912eb46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6"
Nov 02 23:40:12.949800 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:12.949775    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba0cc63-6a7a-474c-b6e4-c411f912eb46-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6\" (UID: \"fba0cc63-6a7a-474c-b6e4-c411f912eb46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6"
Nov 02 23:40:13.003577 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:13.003550    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ldwh\" (UniqueName: \"kubernetes.io/projected/fba0cc63-6a7a-474c-b6e4-c411f912eb46-kube-api-access-4ldwh\") pod \"payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6\" (UID: \"fba0cc63-6a7a-474c-b6e4-c411f912eb46\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6"
Nov 02 23:40:13.048760 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:13.048728    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6"
Nov 02 23:40:13.426349 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:13.426319    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6"]
Nov 02 23:40:13.427141 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:40:13.427116    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba0cc63_6a7a_474c_b6e4_c411f912eb46.slice/crio-b7c69854072fd63d3bde528f76136c5aebe3942e91d7fd111a80ab9079c4dbe5 WatchSource:0}: Error finding container b7c69854072fd63d3bde528f76136c5aebe3942e91d7fd111a80ab9079c4dbe5: Status 404 returned error can't find the container with id b7c69854072fd63d3bde528f76136c5aebe3942e91d7fd111a80ab9079c4dbe5
Nov 02 23:40:14.118369 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:14.118331    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:14.249603 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:14.249570    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6" event={"ID":"fba0cc63-6a7a-474c-b6e4-c411f912eb46","Type":"ContainerStarted","Data":"d66244dc81c334afe93fc8d4ea6e574e4fda691f1ed8665a263b1cdb42b6e6be"}
Nov 02 23:40:14.249603 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:14.249619    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6" event={"ID":"fba0cc63-6a7a-474c-b6e4-c411f912eb46","Type":"ContainerStarted","Data":"b7c69854072fd63d3bde528f76136c5aebe3942e91d7fd111a80ab9079c4dbe5"}
Nov 02 23:40:14.249879 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:14.249684    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6"
Nov 02 23:40:14.304420 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:14.304384    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:15.862576 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:15.862540    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:19.116148 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:19.116113    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:19.274919 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:19.274882    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:19.489067 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:19.488954    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6" podStartSLOduration=7.488939362 podStartE2EDuration="7.488939362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:40:14.360687905 +0000 UTC m=+2853.929042691" watchObservedRunningTime="2026-11-02 23:40:19.488939362 +0000 UTC m=+2859.057294198"
Nov 02 23:40:19.491310 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:19.491284    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj"]
Nov 02 23:40:19.495812 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:19.495793    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj"
Nov 02 23:40:19.529639 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:19.529619    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knd76\" (UniqueName: \"kubernetes.io/projected/edca56c1-37b2-4055-bb5a-d43f3d6cb47a-kube-api-access-knd76\") pod \"payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj\" (UID: \"edca56c1-37b2-4055-bb5a-d43f3d6cb47a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj"
Nov 02 23:40:19.529746 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:19.529677    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edca56c1-37b2-4055-bb5a-d43f3d6cb47a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj\" (UID: \"edca56c1-37b2-4055-bb5a-d43f3d6cb47a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj"
Nov 02 23:40:19.549082 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:19.549040    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj"]
Nov 02 23:40:19.591821 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:19.591777    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-7ca60f-dockercfg-sxtbb\""
Nov 02 23:40:19.592143 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:19.592114    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7ca60f\""
Nov 02 23:40:19.630900 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:19.630853    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edca56c1-37b2-4055-bb5a-d43f3d6cb47a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj\" (UID: \"edca56c1-37b2-4055-bb5a-d43f3d6cb47a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj"
Nov 02 23:40:19.631263 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:19.631241    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knd76\" (UniqueName: \"kubernetes.io/projected/edca56c1-37b2-4055-bb5a-d43f3d6cb47a-kube-api-access-knd76\") pod \"payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj\" (UID: \"edca56c1-37b2-4055-bb5a-d43f3d6cb47a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj"
Nov 02 23:40:19.632570 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:19.632541    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edca56c1-37b2-4055-bb5a-d43f3d6cb47a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj\" (UID: \"edca56c1-37b2-4055-bb5a-d43f3d6cb47a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj"
Nov 02 23:40:19.693384 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:19.693352    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knd76\" (UniqueName: \"kubernetes.io/projected/edca56c1-37b2-4055-bb5a-d43f3d6cb47a-kube-api-access-knd76\") pod \"payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj\" (UID: \"edca56c1-37b2-4055-bb5a-d43f3d6cb47a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj"
Nov 02 23:40:19.809802 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:19.809769    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj"
Nov 02 23:40:20.179201 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:20.179170    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj"]
Nov 02 23:40:20.179667 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:40:20.179644    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedca56c1_37b2_4055_bb5a_d43f3d6cb47a.slice/crio-b2e971caf676b7d11ba3f011ba00a78932b87f10d9476f95f0b2959eb13d2d47 WatchSource:0}: Error finding container b2e971caf676b7d11ba3f011ba00a78932b87f10d9476f95f0b2959eb13d2d47: Status 404 returned error can't find the container with id b2e971caf676b7d11ba3f011ba00a78932b87f10d9476f95f0b2959eb13d2d47
Nov 02 23:40:20.287772 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:20.287739    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj" event={"ID":"edca56c1-37b2-4055-bb5a-d43f3d6cb47a","Type":"ContainerStarted","Data":"6f1b86e19226c1434e57926cf7a2538930933c6fae8775196e862ba76705a352"}
Nov 02 23:40:20.287971 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:20.287798    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj"
Nov 02 23:40:20.287971 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:20.287817    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj" event={"ID":"edca56c1-37b2-4055-bb5a-d43f3d6cb47a","Type":"ContainerStarted","Data":"b2e971caf676b7d11ba3f011ba00a78932b87f10d9476f95f0b2959eb13d2d47"}
Nov 02 23:40:20.399938 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:20.399881    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj" podStartSLOduration=1.3998680430000001 podStartE2EDuration="1.399868043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:40:20.397635061 +0000 UTC m=+2859.965989849" watchObservedRunningTime="2026-11-02 23:40:20.399868043 +0000 UTC m=+2859.968222830"
Nov 02 23:40:21.303465 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:21.303434    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:25.258804 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:25.258770    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6"
Nov 02 23:40:26.794949 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:26.794916    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:28.652524 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:28.652493    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:30.215640 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:30.215595    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:31.297322 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:31.297295    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj"
Nov 02 23:40:31.905533 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:31.905503    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:36.130916 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:36.130876    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:36.810929 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:36.810898    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:36.910965 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:36.910927    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft"]
Nov 02 23:40:37.011590 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:37.011558    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58"]
Nov 02 23:40:37.281122 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:37.281091    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:37.444850 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:37.444816    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:37.830014 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:37.829961    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:41.652995 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:41.652950    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:41.750889 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:40:41.750853    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:40:41.911967 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:41.911888    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft" podUID="89551982-54d3-4006-b91f-8abcc25ed76a" containerName="payload-processing" containerID="cri-o://126dd9d76a5b14151f26451d08776c12e34cdb0cdb0411a0e1c1e2c4e77c47c5" gracePeriod=25
Nov 02 23:40:42.012064 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.012024    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58" podUID="3d4f752c-2b7d-4f18-87a0-44b7315464a4" containerName="payload-pre-processing" containerID="cri-o://515e8bab182d2ad9dfbf424b41ae7503debdc0e8d8e628ae50c8c2acd006f0cc" gracePeriod=25
Nov 02 23:40:42.171448 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.171368    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:42.174144 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.174122    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft"
Nov 02 23:40:42.220023 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.219967    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89551982-54d3-4006-b91f-8abcc25ed76a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89551982-54d3-4006-b91f-8abcc25ed76a-plugins-config-volume\") pod \"89551982-54d3-4006-b91f-8abcc25ed76a\" (UID: \"89551982-54d3-4006-b91f-8abcc25ed76a\") "
Nov 02 23:40:42.220264 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.220061    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89551982-54d3-4006-b91f-8abcc25ed76a-kube-api-access-8cxj8\" (UniqueName: \"kubernetes.io/projected/89551982-54d3-4006-b91f-8abcc25ed76a-kube-api-access-8cxj8\") pod \"89551982-54d3-4006-b91f-8abcc25ed76a\" (UID: \"89551982-54d3-4006-b91f-8abcc25ed76a\") "
Nov 02 23:40:42.220433 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.220411    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89551982-54d3-4006-b91f-8abcc25ed76a-plugins-config-volume" pod "89551982-54d3-4006-b91f-8abcc25ed76a" (UID: "89551982-54d3-4006-b91f-8abcc25ed76a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:40:42.220804 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.220774    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89551982-54d3-4006-b91f-8abcc25ed76a-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:40:42.222916 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.222893    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89551982-54d3-4006-b91f-8abcc25ed76a-kube-api-access-8cxj8" pod "89551982-54d3-4006-b91f-8abcc25ed76a" (UID: "89551982-54d3-4006-b91f-8abcc25ed76a"). InnerVolumeSpecName "kube-api-access-8cxj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:40:42.275352 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.275317    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58"
Nov 02 23:40:42.321222 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.321191    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d4f752c-2b7d-4f18-87a0-44b7315464a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d4f752c-2b7d-4f18-87a0-44b7315464a4-plugins-config-volume\") pod \"3d4f752c-2b7d-4f18-87a0-44b7315464a4\" (UID: \"3d4f752c-2b7d-4f18-87a0-44b7315464a4\") "
Nov 02 23:40:42.321422 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.321243    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4f752c-2b7d-4f18-87a0-44b7315464a4-kube-api-access-mngfs\" (UniqueName: \"kubernetes.io/projected/3d4f752c-2b7d-4f18-87a0-44b7315464a4-kube-api-access-mngfs\") pod \"3d4f752c-2b7d-4f18-87a0-44b7315464a4\" (UID: \"3d4f752c-2b7d-4f18-87a0-44b7315464a4\") "
Nov 02 23:40:42.321682 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.321652    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4f752c-2b7d-4f18-87a0-44b7315464a4-plugins-config-volume" pod "3d4f752c-2b7d-4f18-87a0-44b7315464a4" (UID: "3d4f752c-2b7d-4f18-87a0-44b7315464a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:40:42.321777 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.321669    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cxj8\" (UniqueName: \"kubernetes.io/projected/89551982-54d3-4006-b91f-8abcc25ed76a-kube-api-access-8cxj8\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:40:42.323393 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.323371    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4f752c-2b7d-4f18-87a0-44b7315464a4-kube-api-access-mngfs" pod "3d4f752c-2b7d-4f18-87a0-44b7315464a4" (UID: "3d4f752c-2b7d-4f18-87a0-44b7315464a4"). InnerVolumeSpecName "kube-api-access-mngfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:40:42.336624 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.336600    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:42.423550 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.423469    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d4f752c-2b7d-4f18-87a0-44b7315464a4-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:40:42.423550 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.423495    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mngfs\" (UniqueName: \"kubernetes.io/projected/3d4f752c-2b7d-4f18-87a0-44b7315464a4-kube-api-access-mngfs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:40:42.426192 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.426166    4294 generic.go:356] "Generic (PLEG): container finished" podID="3d4f752c-2b7d-4f18-87a0-44b7315464a4" containerID="515e8bab182d2ad9dfbf424b41ae7503debdc0e8d8e628ae50c8c2acd006f0cc" exitCode=0
Nov 02 23:40:42.426333 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.426202    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58" event={"ID":"3d4f752c-2b7d-4f18-87a0-44b7315464a4","Type":"ContainerDied","Data":"515e8bab182d2ad9dfbf424b41ae7503debdc0e8d8e628ae50c8c2acd006f0cc"}
Nov 02 23:40:42.426333 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.426273    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58" event={"ID":"3d4f752c-2b7d-4f18-87a0-44b7315464a4","Type":"ContainerDied","Data":"fe4b236c86b0fad853d0a6da1744edfbf4e5e34b7ad6653c5f3da16edac7b1ff"}
Nov 02 23:40:42.426333 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.426215    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58"
Nov 02 23:40:42.426333 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.426308    4294 scope.go:122] "RemoveContainer" containerID="515e8bab182d2ad9dfbf424b41ae7503debdc0e8d8e628ae50c8c2acd006f0cc"
Nov 02 23:40:42.427691 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.427654    4294 generic.go:356] "Generic (PLEG): container finished" podID="89551982-54d3-4006-b91f-8abcc25ed76a" containerID="126dd9d76a5b14151f26451d08776c12e34cdb0cdb0411a0e1c1e2c4e77c47c5" exitCode=0
Nov 02 23:40:42.427812 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.427719    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft"
Nov 02 23:40:42.427812 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.427732    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft" event={"ID":"89551982-54d3-4006-b91f-8abcc25ed76a","Type":"ContainerDied","Data":"126dd9d76a5b14151f26451d08776c12e34cdb0cdb0411a0e1c1e2c4e77c47c5"}
Nov 02 23:40:42.427812 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.427772    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft" event={"ID":"89551982-54d3-4006-b91f-8abcc25ed76a","Type":"ContainerDied","Data":"b00266a723a17b61871a1b3f3403374e9f0ba10e5a129f344b9b7daf008c465b"}
Nov 02 23:40:42.441856 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.441175    4294 scope.go:122] "RemoveContainer" containerID="515e8bab182d2ad9dfbf424b41ae7503debdc0e8d8e628ae50c8c2acd006f0cc"
Nov 02 23:40:42.441856 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:40:42.441498    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515e8bab182d2ad9dfbf424b41ae7503debdc0e8d8e628ae50c8c2acd006f0cc\": container with ID starting with 515e8bab182d2ad9dfbf424b41ae7503debdc0e8d8e628ae50c8c2acd006f0cc not found: ID does not exist" containerID="515e8bab182d2ad9dfbf424b41ae7503debdc0e8d8e628ae50c8c2acd006f0cc"
Nov 02 23:40:42.441856 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.441530    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515e8bab182d2ad9dfbf424b41ae7503debdc0e8d8e628ae50c8c2acd006f0cc"} err="failed to get container status \"515e8bab182d2ad9dfbf424b41ae7503debdc0e8d8e628ae50c8c2acd006f0cc\": rpc error: code = NotFound desc = could not find container \"515e8bab182d2ad9dfbf424b41ae7503debdc0e8d8e628ae50c8c2acd006f0cc\": container with ID starting with 515e8bab182d2ad9dfbf424b41ae7503debdc0e8d8e628ae50c8c2acd006f0cc not found: ID does not exist"
Nov 02 23:40:42.441856 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.441554    4294 scope.go:122] "RemoveContainer" containerID="126dd9d76a5b14151f26451d08776c12e34cdb0cdb0411a0e1c1e2c4e77c47c5"
Nov 02 23:40:42.450612 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.450595    4294 scope.go:122] "RemoveContainer" containerID="126dd9d76a5b14151f26451d08776c12e34cdb0cdb0411a0e1c1e2c4e77c47c5"
Nov 02 23:40:42.450876 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:40:42.450856    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"126dd9d76a5b14151f26451d08776c12e34cdb0cdb0411a0e1c1e2c4e77c47c5\": container with ID starting with 126dd9d76a5b14151f26451d08776c12e34cdb0cdb0411a0e1c1e2c4e77c47c5 not found: ID does not exist" containerID="126dd9d76a5b14151f26451d08776c12e34cdb0cdb0411a0e1c1e2c4e77c47c5"
Nov 02 23:40:42.450926 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.450884    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126dd9d76a5b14151f26451d08776c12e34cdb0cdb0411a0e1c1e2c4e77c47c5"} err="failed to get container status \"126dd9d76a5b14151f26451d08776c12e34cdb0cdb0411a0e1c1e2c4e77c47c5\": rpc error: code = NotFound desc = could not find container \"126dd9d76a5b14151f26451d08776c12e34cdb0cdb0411a0e1c1e2c4e77c47c5\": container with ID starting with 126dd9d76a5b14151f26451d08776c12e34cdb0cdb0411a0e1c1e2c4e77c47c5 not found: ID does not exist"
Nov 02 23:40:42.490661 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.490632    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:42.583687 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.583649    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58"]
Nov 02 23:40:42.586788 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.586758    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-9672a1-5df47867ff-fsh58"]
Nov 02 23:40:42.691206 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.691119    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft"]
Nov 02 23:40:42.694047 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:42.694028    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9672a1-58dd79f5c5-9zsft"]
Nov 02 23:40:44.548720 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:44.548687    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d4f752c-2b7d-4f18-87a0-44b7315464a4" path="/var/lib/kubelet/pods/3d4f752c-2b7d-4f18-87a0-44b7315464a4/volumes"
Nov 02 23:40:44.549167 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:44.549151    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89551982-54d3-4006-b91f-8abcc25ed76a" path="/var/lib/kubelet/pods/89551982-54d3-4006-b91f-8abcc25ed76a/volumes"
Nov 02 23:40:47.693645 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:47.693613    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:53.181552 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:53.181506    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:40:58.108530 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:40:58.108500    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:02.642494 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.642465    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"]
Nov 02 23:41:02.643088 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.643074    4294 cpu_manager.go:414] "containerMap: removing container" podUID="89551982-54d3-4006-b91f-8abcc25ed76a" containerName="payload-processing"
Nov 02 23:41:02.643148 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.643089    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89551982-54d3-4006-b91f-8abcc25ed76a" containerName="payload-processing"
Nov 02 23:41:02.643148 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.643121    4294 cpu_manager.go:414] "containerMap: removing container" podUID="3d4f752c-2b7d-4f18-87a0-44b7315464a4" containerName="payload-pre-processing"
Nov 02 23:41:02.643148 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.643126    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4f752c-2b7d-4f18-87a0-44b7315464a4" containerName="payload-pre-processing"
Nov 02 23:41:02.643244 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.643220    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4f752c-2b7d-4f18-87a0-44b7315464a4" containerName="payload-pre-processing"
Nov 02 23:41:02.643244 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.643234    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="89551982-54d3-4006-b91f-8abcc25ed76a" containerName="payload-processing"
Nov 02 23:41:02.648283 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.648259    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"
Nov 02 23:41:02.685803 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.685778    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b42173f0-7dde-4729-b3ae-ba90082ca6d8-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94\" (UID: \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"
Nov 02 23:41:02.685905 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.685851    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b42173f0-7dde-4729-b3ae-ba90082ca6d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94\" (UID: \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"
Nov 02 23:41:02.685959 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.685938    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42173f0-7dde-4729-b3ae-ba90082ca6d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94\" (UID: \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"
Nov 02 23:41:02.686019 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.685972    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jdlc\" (UniqueName: \"kubernetes.io/projected/b42173f0-7dde-4729-b3ae-ba90082ca6d8-kube-api-access-5jdlc\") pod \"maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94\" (UID: \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"
Nov 02 23:41:02.699508 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.699487    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"]
Nov 02 23:41:02.741884 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.741855    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-c25940-serving-cert\""
Nov 02 23:41:02.787185 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.787149    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b42173f0-7dde-4729-b3ae-ba90082ca6d8-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94\" (UID: \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"
Nov 02 23:41:02.787357 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.787282    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b42173f0-7dde-4729-b3ae-ba90082ca6d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94\" (UID: \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"
Nov 02 23:41:02.787427 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.787359    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42173f0-7dde-4729-b3ae-ba90082ca6d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94\" (UID: \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"
Nov 02 23:41:02.787427 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.787393    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdlc\" (UniqueName: \"kubernetes.io/projected/b42173f0-7dde-4729-b3ae-ba90082ca6d8-kube-api-access-5jdlc\") pod \"maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94\" (UID: \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"
Nov 02 23:41:02.787702 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:41:02.787683    4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-serving-cert: secret "maas-api-e2e-shared-b-w1-c25940-serving-cert" not found
Nov 02 23:41:02.787831 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:41:02.787820    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b42173f0-7dde-4729-b3ae-ba90082ca6d8-maas-api-tls podName:b42173f0-7dde-4729-b3ae-ba90082ca6d8 nodeName:}" failed. No retries permitted until 2026-11-02 23:41:03.287801187 +0000 UTC m=+2902.856155958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b42173f0-7dde-4729-b3ae-ba90082ca6d8-maas-api-tls") pod "maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94" (UID: "b42173f0-7dde-4729-b3ae-ba90082ca6d8") : secret "maas-api-e2e-shared-b-w1-c25940-serving-cert" not found
Nov 02 23:41:02.788561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.788537    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42173f0-7dde-4729-b3ae-ba90082ca6d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94\" (UID: \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"
Nov 02 23:41:02.790326 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.790302    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b42173f0-7dde-4729-b3ae-ba90082ca6d8-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94\" (UID: \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"
Nov 02 23:41:02.842298 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.842277    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jdlc\" (UniqueName: \"kubernetes.io/projected/b42173f0-7dde-4729-b3ae-ba90082ca6d8-kube-api-access-5jdlc\") pod \"maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94\" (UID: \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"
Nov 02 23:41:02.864822 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:02.864798    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:03.034925 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:03.034895    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:03.194136 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:03.194105    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:03.293202 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:03.293120    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b42173f0-7dde-4729-b3ae-ba90082ca6d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94\" (UID: \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"
Nov 02 23:41:03.295515 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:03.295490    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b42173f0-7dde-4729-b3ae-ba90082ca6d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94\" (UID: \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"
Nov 02 23:41:03.562378 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:03.562296    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"
Nov 02 23:41:03.939057 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:41:03.939021    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42173f0_7dde_4729_b3ae_ba90082ca6d8.slice/crio-989279a5162c5ec69a07effe1afae0fb1495d1e34e62f81078ebab53ebcf8485 WatchSource:0}: Error finding container 989279a5162c5ec69a07effe1afae0fb1495d1e34e62f81078ebab53ebcf8485: Status 404 returned error can't find the container with id 989279a5162c5ec69a07effe1afae0fb1495d1e34e62f81078ebab53ebcf8485
Nov 02 23:41:03.939415 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:03.939099    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"]
Nov 02 23:41:04.567020 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:04.566987    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94" event={"ID":"b42173f0-7dde-4729-b3ae-ba90082ca6d8","Type":"ContainerStarted","Data":"989279a5162c5ec69a07effe1afae0fb1495d1e34e62f81078ebab53ebcf8485"}
Nov 02 23:41:05.574716 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:05.574678    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94" event={"ID":"b42173f0-7dde-4729-b3ae-ba90082ca6d8","Type":"ContainerStarted","Data":"450153df7da90af38c8bd61e559f86aa544e6a03baffad7fd7173e09c4fe6d23"}
Nov 02 23:41:05.575187 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:05.574732    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"
Nov 02 23:41:05.688550 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:05.688489    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94" podStartSLOduration=2.811385822 podStartE2EDuration="3.68847024s" totalImagesPullingTime="877.084418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:41:05.68431953 +0000 UTC m=+2905.252674319" watchObservedRunningTime="2026-11-02 23:41:05.68847024 +0000 UTC m=+2905.256825028"
Nov 02 23:41:08.077492 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:08.077444    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:11.588189 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:11.588134    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"
Nov 02 23:41:11.767829 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:41:11.767789    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:41:12.980338 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:12.980303    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:17.495251 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:17.495209    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"]
Nov 02 23:41:17.499860 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:17.499837    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"
Nov 02 23:41:17.552327 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:17.552301    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"]
Nov 02 23:41:17.572451 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:17.572409    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c80137-cc6a-48e5-9e02-184ee47f56da-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt\" (UID: \"00c80137-cc6a-48e5-9e02-184ee47f56da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"
Nov 02 23:41:17.572712 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:17.572688    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwr79\" (UniqueName: \"kubernetes.io/projected/00c80137-cc6a-48e5-9e02-184ee47f56da-kube-api-access-hwr79\") pod \"maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt\" (UID: \"00c80137-cc6a-48e5-9e02-184ee47f56da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"
Nov 02 23:41:17.572773 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:17.572730    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c80137-cc6a-48e5-9e02-184ee47f56da-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt\" (UID: \"00c80137-cc6a-48e5-9e02-184ee47f56da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"
Nov 02 23:41:17.572871 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:17.572852    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c80137-cc6a-48e5-9e02-184ee47f56da-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt\" (UID: \"00c80137-cc6a-48e5-9e02-184ee47f56da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"
Nov 02 23:41:17.595947 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:17.595926    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-a51378-serving-cert\""
Nov 02 23:41:17.673937 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:17.673904    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c80137-cc6a-48e5-9e02-184ee47f56da-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt\" (UID: \"00c80137-cc6a-48e5-9e02-184ee47f56da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"
Nov 02 23:41:17.674091 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:17.674081    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwr79\" (UniqueName: \"kubernetes.io/projected/00c80137-cc6a-48e5-9e02-184ee47f56da-kube-api-access-hwr79\") pod \"maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt\" (UID: \"00c80137-cc6a-48e5-9e02-184ee47f56da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"
Nov 02 23:41:17.674148 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:17.674107    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c80137-cc6a-48e5-9e02-184ee47f56da-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt\" (UID: \"00c80137-cc6a-48e5-9e02-184ee47f56da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"
Nov 02 23:41:17.674185 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:17.674159    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c80137-cc6a-48e5-9e02-184ee47f56da-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt\" (UID: \"00c80137-cc6a-48e5-9e02-184ee47f56da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"
Nov 02 23:41:17.674281 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:41:17.674269    4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-serving-cert: secret "maas-api-e2e-shared-b-w4-a51378-serving-cert" not found
Nov 02 23:41:17.674336 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:41:17.674320    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c80137-cc6a-48e5-9e02-184ee47f56da-maas-api-tls podName:00c80137-cc6a-48e5-9e02-184ee47f56da nodeName:}" failed. No retries permitted until 2026-11-02 23:41:18.17430548 +0000 UTC m=+2917.742660245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00c80137-cc6a-48e5-9e02-184ee47f56da-maas-api-tls") pod "maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt" (UID: "00c80137-cc6a-48e5-9e02-184ee47f56da") : secret "maas-api-e2e-shared-b-w4-a51378-serving-cert" not found
Nov 02 23:41:17.674759 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:17.674741    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c80137-cc6a-48e5-9e02-184ee47f56da-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt\" (UID: \"00c80137-cc6a-48e5-9e02-184ee47f56da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"
Nov 02 23:41:17.676297 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:17.676278    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c80137-cc6a-48e5-9e02-184ee47f56da-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt\" (UID: \"00c80137-cc6a-48e5-9e02-184ee47f56da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"
Nov 02 23:41:17.728828 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:17.728793    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwr79\" (UniqueName: \"kubernetes.io/projected/00c80137-cc6a-48e5-9e02-184ee47f56da-kube-api-access-hwr79\") pod \"maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt\" (UID: \"00c80137-cc6a-48e5-9e02-184ee47f56da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"
Nov 02 23:41:17.749268 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:17.749199    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:17.909866 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:17.909833    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:18.073479 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:18.073401    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:18.180438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:18.180409    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c80137-cc6a-48e5-9e02-184ee47f56da-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt\" (UID: \"00c80137-cc6a-48e5-9e02-184ee47f56da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"
Nov 02 23:41:18.182728 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:18.182702    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c80137-cc6a-48e5-9e02-184ee47f56da-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt\" (UID: \"00c80137-cc6a-48e5-9e02-184ee47f56da\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"
Nov 02 23:41:18.413480 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:18.413403    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"
Nov 02 23:41:18.781070 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:41:18.781041    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c80137_cc6a_48e5_9e02_184ee47f56da.slice/crio-4ddcf8bc46df3a4e9858219483d162ed8c297be5804c823f084f4349eddae038 WatchSource:0}: Error finding container 4ddcf8bc46df3a4e9858219483d162ed8c297be5804c823f084f4349eddae038: Status 404 returned error can't find the container with id 4ddcf8bc46df3a4e9858219483d162ed8c297be5804c823f084f4349eddae038
Nov 02 23:41:18.781520 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:18.781331    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"]
Nov 02 23:41:19.671063 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:19.671025    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt" event={"ID":"00c80137-cc6a-48e5-9e02-184ee47f56da","Type":"ContainerStarted","Data":"4ddcf8bc46df3a4e9858219483d162ed8c297be5804c823f084f4349eddae038"}
Nov 02 23:41:20.678940 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:20.678887    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt" event={"ID":"00c80137-cc6a-48e5-9e02-184ee47f56da","Type":"ContainerStarted","Data":"30db606424c8fb22e8bbba46b97472afba6c26d51490da1f31bd0b10938f0850"}
Nov 02 23:41:20.679393 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:20.678973    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"
Nov 02 23:41:20.790781 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:20.790729    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt" podStartSLOduration=2.868892992 podStartE2EDuration="3.790715839s" totalImagesPullingTime="921.822847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:41:20.787204367 +0000 UTC m=+2920.355559146" watchObservedRunningTime="2026-11-02 23:41:20.790715839 +0000 UTC m=+2920.359070625"
Nov 02 23:41:22.291272 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:22.291240    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:24.074222 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:24.074127    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:26.690423 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:26.690392    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"
Nov 02 23:41:28.807179 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:28.807148    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:28.970228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:28.970188    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:29.134337 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:29.134255    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:31.315935 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:31.315836    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:31.529879 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:31.529841    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:33.001328 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:33.001284    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:33.539522 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:33.539494    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:34.299072 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:34.299037    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:34.471016 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:34.468446    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:38.151467 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:38.151430    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:39.484533 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:39.484501    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:40.323279 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:40.323247    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:40.654063 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:40.654037    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph"]
Nov 02 23:41:40.834721 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:40.834689    4294 generic.go:356] "Generic (PLEG): container finished" podID="30dbc36b-dc80-47c8-a95a-5ad407f121a1" containerID="97abd1623f7b267f2e098831e569610e8eef2a932a8321904ca8eabd028014b0" exitCode=0
Nov 02 23:41:40.834909 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:40.834763    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb" event={"ID":"30dbc36b-dc80-47c8-a95a-5ad407f121a1","Type":"ContainerDied","Data":"97abd1623f7b267f2e098831e569610e8eef2a932a8321904ca8eabd028014b0"}
Nov 02 23:41:40.835238 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:40.835220    4294 scope.go:122] "RemoveContainer" containerID="97abd1623f7b267f2e098831e569610e8eef2a932a8321904ca8eabd028014b0"
Nov 02 23:41:40.836820 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:40.836793    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:41:40.836919 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:40.836872    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:41:40.837015 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:40.836922    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:41:41.034664 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:41.034630    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:41.197740 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:41.197650    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:41.751141 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:41:41.751108    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:41:41.842609 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:41.842576    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb" event={"ID":"30dbc36b-dc80-47c8-a95a-5ad407f121a1","Type":"ContainerStarted","Data":"6080a5cf2e47e6e77e9614082203d545a4a9dc1cccc789f7d6b95d985c6ca12c"}
Nov 02 23:41:42.708882 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:42.708849    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:41:42.709076 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:42.709040    4294 patch_prober.go:29] interesting pod/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 02 23:41:42.709133 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:42.709087    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb" podUID="30dbc36b-dc80-47c8-a95a-5ad407f121a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 02 23:41:43.709590 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:43.709551    4294 patch_prober.go:29] interesting pod/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 02 23:41:43.710046 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:43.709619    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb" podUID="30dbc36b-dc80-47c8-a95a-5ad407f121a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 02 23:41:44.575958 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:44.575911    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:44.709650 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:44.709612    4294 patch_prober.go:29] interesting pod/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 02 23:41:44.710049 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:44.709674    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb" podUID="30dbc36b-dc80-47c8-a95a-5ad407f121a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 02 23:41:45.667009 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:45.661033    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph" podUID="d2b96272-5bca-4475-84bb-6022ec383f6f" containerName="payload-processing" containerID="cri-o://83146dbf02ece98d6fb15e076922b976a287ed823f0734be809f3a1e6fdd7c0a" gracePeriod=25
Nov 02 23:41:45.709421 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:45.709392    4294 patch_prober.go:29] interesting pod/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 02 23:41:45.709516 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:45.709455    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb" podUID="30dbc36b-dc80-47c8-a95a-5ad407f121a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 02 23:41:45.834634 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:45.834594    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:45.903619 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:45.903287    4294 generic.go:356] "Generic (PLEG): container finished" podID="d2b96272-5bca-4475-84bb-6022ec383f6f" containerID="83146dbf02ece98d6fb15e076922b976a287ed823f0734be809f3a1e6fdd7c0a" exitCode=0
Nov 02 23:41:45.903619 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:45.903502    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph" event={"ID":"d2b96272-5bca-4475-84bb-6022ec383f6f","Type":"ContainerDied","Data":"83146dbf02ece98d6fb15e076922b976a287ed823f0734be809f3a1e6fdd7c0a"}
Nov 02 23:41:45.988413 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:45.988389    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph"
Nov 02 23:41:46.176002 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:46.166575    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b96272-5bca-4475-84bb-6022ec383f6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b96272-5bca-4475-84bb-6022ec383f6f-plugins-config-volume\") pod \"d2b96272-5bca-4475-84bb-6022ec383f6f\" (UID: \"d2b96272-5bca-4475-84bb-6022ec383f6f\") "
Nov 02 23:41:46.179240 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:46.179210    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b96272-5bca-4475-84bb-6022ec383f6f-kube-api-access-69kwp\" (UniqueName: \"kubernetes.io/projected/d2b96272-5bca-4475-84bb-6022ec383f6f-kube-api-access-69kwp\") pod \"d2b96272-5bca-4475-84bb-6022ec383f6f\" (UID: \"d2b96272-5bca-4475-84bb-6022ec383f6f\") "
Nov 02 23:41:46.180751 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:46.180721    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b96272-5bca-4475-84bb-6022ec383f6f-plugins-config-volume" pod "d2b96272-5bca-4475-84bb-6022ec383f6f" (UID: "d2b96272-5bca-4475-84bb-6022ec383f6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:41:46.183177 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:46.183149    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b96272-5bca-4475-84bb-6022ec383f6f-kube-api-access-69kwp" pod "d2b96272-5bca-4475-84bb-6022ec383f6f" (UID: "d2b96272-5bca-4475-84bb-6022ec383f6f"). InnerVolumeSpecName "kube-api-access-69kwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:41:46.281223 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:46.281180    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69kwp\" (UniqueName: \"kubernetes.io/projected/d2b96272-5bca-4475-84bb-6022ec383f6f-kube-api-access-69kwp\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:41:46.281223 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:46.281220    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b96272-5bca-4475-84bb-6022ec383f6f-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:41:46.708920 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:46.708887    4294 patch_prober.go:29] interesting pod/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 02 23:41:46.709103 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:46.708940    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb" podUID="30dbc36b-dc80-47c8-a95a-5ad407f121a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 02 23:41:46.910390 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:46.910309    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph"
Nov 02 23:41:46.910390 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:46.910340    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph" event={"ID":"d2b96272-5bca-4475-84bb-6022ec383f6f","Type":"ContainerDied","Data":"df3ef12dcada36d470f2ee10ecb8e43a045a381b62b65d60163097db80b0afff"}
Nov 02 23:41:46.910867 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:46.910404    4294 scope.go:122] "RemoveContainer" containerID="83146dbf02ece98d6fb15e076922b976a287ed823f0734be809f3a1e6fdd7c0a"
Nov 02 23:41:47.069452 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:47.069417    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph"]
Nov 02 23:41:47.117000 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:47.116948    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b1be95-95f9ccdb5-hvxph"]
Nov 02 23:41:47.714142 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:47.714113    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:41:47.714536 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:47.714512    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:41:47.715063 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:47.715045    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 02 23:41:48.548194 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:48.548161    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b96272-5bca-4475-84bb-6022ec383f6f" path="/var/lib/kubelet/pods/d2b96272-5bca-4475-84bb-6022ec383f6f/volumes"
Nov 02 23:41:49.563943 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:49.563906    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:54.492162 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:54.492130    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:41:59.587495 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:41:59.587425    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:04.615232 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:04.615200    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:09.738943 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:09.738893    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:09.901992 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:09.901950    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:11.752883 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:42:11.752065    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:42:14.246199 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.246159    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"]
Nov 02 23:42:14.247039 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.247017    4294 cpu_manager.go:414] "containerMap: removing container" podUID="d2b96272-5bca-4475-84bb-6022ec383f6f" containerName="payload-processing"
Nov 02 23:42:14.247039 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.247039    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b96272-5bca-4475-84bb-6022ec383f6f" containerName="payload-processing"
Nov 02 23:42:14.247237 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.247223    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b96272-5bca-4475-84bb-6022ec383f6f" containerName="payload-processing"
Nov 02 23:42:14.249491 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.249472    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:14.300207 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.300163    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-c5148c\"/\"openshift-service-ca.crt\""
Nov 02 23:42:14.311212 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.311177    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"]
Nov 02 23:42:14.318682 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.318656    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-c5148c\"/\"kube-root-ca.crt\""
Nov 02 23:42:14.346867 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.346838    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-c5148c\"/\"default-dockercfg-mv8gj\""
Nov 02 23:42:14.346867 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.346841    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-c5148c\"/\"auth-test-model-c5148c-kserve-self-signed-certs\""
Nov 02 23:42:14.367225 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.367198    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:14.424206 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.424173    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-tmp-dir\") pod \"auth-test-model-c5148c-kserve-6c96748cf7-h78vh\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") " pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:14.424206 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.424219    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjgj4\" (UniqueName: \"kubernetes.io/projected/94191650-b4f0-4fe2-ad32-cf98558fd894-kube-api-access-qjgj4\") pod \"auth-test-model-c5148c-kserve-6c96748cf7-h78vh\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") " pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:14.424451 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.424347    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-dshm\") pod \"auth-test-model-c5148c-kserve-6c96748cf7-h78vh\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") " pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:14.424503 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.424469    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94191650-b4f0-4fe2-ad32-cf98558fd894-tls-certs\") pod \"auth-test-model-c5148c-kserve-6c96748cf7-h78vh\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") " pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:14.424567 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.424542    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-home\") pod \"auth-test-model-c5148c-kserve-6c96748cf7-h78vh\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") " pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:14.424604 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.424591    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-model-cache\") pod \"auth-test-model-c5148c-kserve-6c96748cf7-h78vh\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") " pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:14.525673 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.525588    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94191650-b4f0-4fe2-ad32-cf98558fd894-tls-certs\") pod \"auth-test-model-c5148c-kserve-6c96748cf7-h78vh\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") " pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:14.525673 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.525641    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-home\") pod \"auth-test-model-c5148c-kserve-6c96748cf7-h78vh\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") " pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:14.525673 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.525660    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-model-cache\") pod \"auth-test-model-c5148c-kserve-6c96748cf7-h78vh\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") " pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:14.525972 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.525714    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-tmp-dir\") pod \"auth-test-model-c5148c-kserve-6c96748cf7-h78vh\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") " pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:14.525972 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.525752    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjgj4\" (UniqueName: \"kubernetes.io/projected/94191650-b4f0-4fe2-ad32-cf98558fd894-kube-api-access-qjgj4\") pod \"auth-test-model-c5148c-kserve-6c96748cf7-h78vh\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") " pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:14.525972 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.525848    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-dshm\") pod \"auth-test-model-c5148c-kserve-6c96748cf7-h78vh\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") " pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:14.526202 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.526155    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-home\") pod \"auth-test-model-c5148c-kserve-6c96748cf7-h78vh\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") " pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:14.526256 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.526202    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-model-cache\") pod \"auth-test-model-c5148c-kserve-6c96748cf7-h78vh\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") " pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:14.526505 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.526471    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-tmp-dir\") pod \"auth-test-model-c5148c-kserve-6c96748cf7-h78vh\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") " pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:14.528205 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.528181    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-dshm\") pod \"auth-test-model-c5148c-kserve-6c96748cf7-h78vh\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") " pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:14.528811 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.528790    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94191650-b4f0-4fe2-ad32-cf98558fd894-tls-certs\") pod \"auth-test-model-c5148c-kserve-6c96748cf7-h78vh\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") " pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:14.593050 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.593022    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjgj4\" (UniqueName: \"kubernetes.io/projected/94191650-b4f0-4fe2-ad32-cf98558fd894-kube-api-access-qjgj4\") pod \"auth-test-model-c5148c-kserve-6c96748cf7-h78vh\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") " pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:14.727109 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.727056    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:14.860462 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:14.860373    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:15.228964 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:15.228941    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"]
Nov 02 23:42:15.229302 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:42:15.229278    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94191650_b4f0_4fe2_ad32_cf98558fd894.slice/crio-aa9b5c07cce41d765b313c0a6090327c2476c047e6141fbfe94c849286917728 WatchSource:0}: Error finding container aa9b5c07cce41d765b313c0a6090327c2476c047e6141fbfe94c849286917728: Status 404 returned error can't find the container with id aa9b5c07cce41d765b313c0a6090327c2476c047e6141fbfe94c849286917728
Nov 02 23:42:16.098055 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:16.098023    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh" event={"ID":"94191650-b4f0-4fe2-ad32-cf98558fd894","Type":"ContainerStarted","Data":"677ff13534b4142308b2bbdec1cd8b084dc7d05c85a3095bde0369ccc47eaee3"}
Nov 02 23:42:16.098452 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:16.098062    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh" event={"ID":"94191650-b4f0-4fe2-ad32-cf98558fd894","Type":"ContainerStarted","Data":"aa9b5c07cce41d765b313c0a6090327c2476c047e6141fbfe94c849286917728"}
Nov 02 23:42:16.208245 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:16.208195    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh" podStartSLOduration=2.208183122 podStartE2EDuration="2.208183122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:42:16.207929114 +0000 UTC m=+2975.776283902" watchObservedRunningTime="2026-11-02 23:42:16.208183122 +0000 UTC m=+2975.776537908"
Nov 02 23:42:16.860711 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:16.860675    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:19.172125 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:19.172095    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:23.753416 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:23.753384    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:24.861049 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:24.861011    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:24.873437 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:24.873413    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:25.166684 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:25.166604    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:42:28.574262 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:28.574222    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:30.434141 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:30.434109    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:32.525728 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:32.525696    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:32.686203 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:32.686172    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:33.156687 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:33.156650    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:37.666912 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:37.666876    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:38.473842 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:38.473813    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:39.352132 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:39.352093    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:41.580088 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:41.580048    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:41.749393 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:42:41.749351    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:42:42.453621 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:42.453588    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:47.052368 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:47.052312    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:51.064339 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:51.064302    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:51.108315 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:51.108280    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"]
Nov 02 23:42:51.108517 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:51.108493    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds" podUID="a80f7343-30ef-4b44-9e28-c3c7a9e80952" containerName="maas-api" containerID="cri-o://327d3030dac889df6ee35645de7ee64df10569e3c4f85b82e42ed532edffecc5" gracePeriod=30
Nov 02 23:42:51.230111 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:51.230082    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:51.321581 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:51.321475    4294 generic.go:356] "Generic (PLEG): container finished" podID="a80f7343-30ef-4b44-9e28-c3c7a9e80952" containerID="327d3030dac889df6ee35645de7ee64df10569e3c4f85b82e42ed532edffecc5" exitCode=0
Nov 02 23:42:51.321581 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:51.321553    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds" event={"ID":"a80f7343-30ef-4b44-9e28-c3c7a9e80952","Type":"ContainerDied","Data":"327d3030dac889df6ee35645de7ee64df10569e3c4f85b82e42ed532edffecc5"}
Nov 02 23:42:51.381900 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:51.381872    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"
Nov 02 23:42:51.445740 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:51.445710    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a80f7343-30ef-4b44-9e28-c3c7a9e80952-kube-api-access-m9q9f\" (UniqueName: \"kubernetes.io/projected/a80f7343-30ef-4b44-9e28-c3c7a9e80952-kube-api-access-m9q9f\") pod \"a80f7343-30ef-4b44-9e28-c3c7a9e80952\" (UID: \"a80f7343-30ef-4b44-9e28-c3c7a9e80952\") "
Nov 02 23:42:51.445907 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:51.445828    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a80f7343-30ef-4b44-9e28-c3c7a9e80952-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80f7343-30ef-4b44-9e28-c3c7a9e80952-openshift-service-ca\") pod \"a80f7343-30ef-4b44-9e28-c3c7a9e80952\" (UID: \"a80f7343-30ef-4b44-9e28-c3c7a9e80952\") "
Nov 02 23:42:51.445907 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:51.445889    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a80f7343-30ef-4b44-9e28-c3c7a9e80952-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a80f7343-30ef-4b44-9e28-c3c7a9e80952-metrics-certs\") pod \"a80f7343-30ef-4b44-9e28-c3c7a9e80952\" (UID: \"a80f7343-30ef-4b44-9e28-c3c7a9e80952\") "
Nov 02 23:42:51.446024 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:51.446006    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a80f7343-30ef-4b44-9e28-c3c7a9e80952-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a80f7343-30ef-4b44-9e28-c3c7a9e80952-maas-api-tls\") pod \"a80f7343-30ef-4b44-9e28-c3c7a9e80952\" (UID: \"a80f7343-30ef-4b44-9e28-c3c7a9e80952\") "
Nov 02 23:42:51.446254 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:51.446229    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80f7343-30ef-4b44-9e28-c3c7a9e80952-openshift-service-ca" pod "a80f7343-30ef-4b44-9e28-c3c7a9e80952" (UID: "a80f7343-30ef-4b44-9e28-c3c7a9e80952"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:42:51.446520 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:51.446486    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80f7343-30ef-4b44-9e28-c3c7a9e80952-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:42:51.448036 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:51.448012    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a80f7343-30ef-4b44-9e28-c3c7a9e80952-metrics-certs" pod "a80f7343-30ef-4b44-9e28-c3c7a9e80952" (UID: "a80f7343-30ef-4b44-9e28-c3c7a9e80952"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:42:51.448137 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:51.448012    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80f7343-30ef-4b44-9e28-c3c7a9e80952-kube-api-access-m9q9f" pod "a80f7343-30ef-4b44-9e28-c3c7a9e80952" (UID: "a80f7343-30ef-4b44-9e28-c3c7a9e80952"). InnerVolumeSpecName "kube-api-access-m9q9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:42:51.448137 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:51.448045    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a80f7343-30ef-4b44-9e28-c3c7a9e80952-maas-api-tls" pod "a80f7343-30ef-4b44-9e28-c3c7a9e80952" (UID: "a80f7343-30ef-4b44-9e28-c3c7a9e80952"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:42:51.547799 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:51.547760    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9q9f\" (UniqueName: \"kubernetes.io/projected/a80f7343-30ef-4b44-9e28-c3c7a9e80952-kube-api-access-m9q9f\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:42:51.547799 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:51.547791    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a80f7343-30ef-4b44-9e28-c3c7a9e80952-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:42:51.547799 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:51.547805    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a80f7343-30ef-4b44-9e28-c3c7a9e80952-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:42:52.331877 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:52.331843    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"
Nov 02 23:42:52.331877 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:52.331868    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds" event={"ID":"a80f7343-30ef-4b44-9e28-c3c7a9e80952","Type":"ContainerDied","Data":"135a6f1d30dc00aa3b8b1df8307a23f89da7cd621357411319e3ad6d83fbad3e"}
Nov 02 23:42:52.332454 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:52.331930    4294 scope.go:122] "RemoveContainer" containerID="327d3030dac889df6ee35645de7ee64df10569e3c4f85b82e42ed532edffecc5"
Nov 02 23:42:52.490931 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:52.490897    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"]
Nov 02 23:42:52.538413 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:52.538374    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-b1be95-84d57dd5b6-twvds"]
Nov 02 23:42:52.549066 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:52.549036    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a80f7343-30ef-4b44-9e28-c3c7a9e80952" path="/var/lib/kubelet/pods/a80f7343-30ef-4b44-9e28-c3c7a9e80952/volumes"
Nov 02 23:42:52.943381 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:52.943338    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:42:57.433149 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:42:57.433065    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:00.899356 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:00.899313    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:01.128934 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:01.128898    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:02.096457 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:02.096417    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:02.403642 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:02.403548    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:02.879017 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:02.878962    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:07.078320 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:07.078287    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:10.930609 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:10.930572    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:11.580613 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:11.580577    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:11.757819 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:43:11.757778    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:43:11.777853 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:11.777820    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:12.166437 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:12.166400    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:14.190409 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.190374    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"]
Nov 02 23:43:14.191255 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.191231    4294 cpu_manager.go:414] "containerMap: removing container" podUID="a80f7343-30ef-4b44-9e28-c3c7a9e80952" containerName="maas-api"
Nov 02 23:43:14.191369 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.191256    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a80f7343-30ef-4b44-9e28-c3c7a9e80952" containerName="maas-api"
Nov 02 23:43:14.191441 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.191414    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a80f7343-30ef-4b44-9e28-c3c7a9e80952" containerName="maas-api"
Nov 02 23:43:14.194724 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.194704    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:14.235437 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.235408    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-tls-certs\") pod \"auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:14.235612 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.235480    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-model-cache\") pod \"auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:14.235612 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.235592    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-dshm\") pod \"auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:14.235738 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.235637    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-home\") pod \"auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:14.235738 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.235724    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48vz9\" (UniqueName: \"kubernetes.io/projected/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-kube-api-access-48vz9\") pod \"auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:14.235865 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.235773    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-tmp-dir\") pod \"auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:14.292619 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.292585    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-fe63ac\"/\"default-dockercfg-tmqkp\""
Nov 02 23:43:14.292619 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.292612    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-fe63ac\"/\"auth-test-model-fe63ac-kserve-self-signed-certs\""
Nov 02 23:43:14.292844 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.292585    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-fe63ac\"/\"kube-root-ca.crt\""
Nov 02 23:43:14.292844 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.292636    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-fe63ac\"/\"openshift-service-ca.crt\""
Nov 02 23:43:14.296122 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.296098    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"]
Nov 02 23:43:14.323932 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.323877    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:14.344273 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.344238    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-model-cache\") pod \"auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:14.344589 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.344563    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-dshm\") pod \"auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:14.344819 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.344800    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-home\") pod \"auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:14.347180 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.347155    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48vz9\" (UniqueName: \"kubernetes.io/projected/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-kube-api-access-48vz9\") pod \"auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:14.347297 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.347265    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-tmp-dir\") pod \"auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:14.347417 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.347324    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-tls-certs\") pod \"auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:14.347839 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.344723    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-model-cache\") pod \"auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:14.347950 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.346585    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-home\") pod \"auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:14.348159 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.348137    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-tmp-dir\") pod \"auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:14.353612 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.353593    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-dshm\") pod \"auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:14.354260 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.354243    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-tls-certs\") pod \"auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:14.403741 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.403714    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48vz9\" (UniqueName: \"kubernetes.io/projected/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-kube-api-access-48vz9\") pod \"auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") " pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:14.506640 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.506612    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:14.876678 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:43:14.876648    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda96bffa8_dbcd_4539_b494_b9ac3cc80dc1.slice/crio-971788d260e297488d0da7f98a5b23d7240db61ecc8f2252994f69e2ce450cef WatchSource:0}: Error finding container 971788d260e297488d0da7f98a5b23d7240db61ecc8f2252994f69e2ce450cef: Status 404 returned error can't find the container with id 971788d260e297488d0da7f98a5b23d7240db61ecc8f2252994f69e2ce450cef
Nov 02 23:43:14.876909 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:14.876889    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"]
Nov 02 23:43:15.502734 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:15.502702    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r" event={"ID":"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1","Type":"ContainerStarted","Data":"9d8ea634e41310eeab057abe3e532b668834f6a5d27ee5727dfbd689f6bed475"}
Nov 02 23:43:15.503137 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:15.502742    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r" event={"ID":"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1","Type":"ContainerStarted","Data":"971788d260e297488d0da7f98a5b23d7240db61ecc8f2252994f69e2ce450cef"}
Nov 02 23:43:15.507033 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:15.507019    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:15.612120 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:15.612066    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r" podStartSLOduration=1.6120494829999998 podStartE2EDuration="1.612049483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:43:15.611446597 +0000 UTC m=+3035.179801384" watchObservedRunningTime="2026-11-02 23:43:15.612049483 +0000 UTC m=+3035.180404286"
Nov 02 23:43:16.127449 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:16.127416    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:16.307685 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:16.307652    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:16.487044 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:16.486916    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:20.005360 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:20.005323    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:20.159386 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:20.159347    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:20.378254 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:20.378167    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4"]
Nov 02 23:43:20.382772 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:20.382749    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4"
Nov 02 23:43:20.422071 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:20.422042    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch7r5\" (UniqueName: \"kubernetes.io/projected/28664c33-54e8-4ced-9a16-dc46980a8bf5-kube-api-access-ch7r5\") pod \"payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4\" (UID: \"28664c33-54e8-4ced-9a16-dc46980a8bf5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4"
Nov 02 23:43:20.422350 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:20.422318    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28664c33-54e8-4ced-9a16-dc46980a8bf5-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4\" (UID: \"28664c33-54e8-4ced-9a16-dc46980a8bf5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4"
Nov 02 23:43:20.435153 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:20.435127    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4"]
Nov 02 23:43:20.478282 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:20.478251    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-52dd59-dockercfg-m8mlw\""
Nov 02 23:43:20.495285 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:20.488797    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-52dd59\""
Nov 02 23:43:20.524101 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:20.524064    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28664c33-54e8-4ced-9a16-dc46980a8bf5-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4\" (UID: \"28664c33-54e8-4ced-9a16-dc46980a8bf5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4"
Nov 02 23:43:20.524281 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:20.524143    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch7r5\" (UniqueName: \"kubernetes.io/projected/28664c33-54e8-4ced-9a16-dc46980a8bf5-kube-api-access-ch7r5\") pod \"payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4\" (UID: \"28664c33-54e8-4ced-9a16-dc46980a8bf5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4"
Nov 02 23:43:20.524860 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:20.524837    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28664c33-54e8-4ced-9a16-dc46980a8bf5-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4\" (UID: \"28664c33-54e8-4ced-9a16-dc46980a8bf5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4"
Nov 02 23:43:20.579188 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:20.579158    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch7r5\" (UniqueName: \"kubernetes.io/projected/28664c33-54e8-4ced-9a16-dc46980a8bf5-kube-api-access-ch7r5\") pod \"payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4\" (UID: \"28664c33-54e8-4ced-9a16-dc46980a8bf5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4"
Nov 02 23:43:20.697464 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:20.697389    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4"
Nov 02 23:43:21.070107 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:21.070076    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4"]
Nov 02 23:43:21.070596 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:43:21.070196    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28664c33_54e8_4ced_9a16_dc46980a8bf5.slice/crio-87f515c5d59ad95e0bc6fe1689756f345662d5fe67277a8c87dbeeced93ca021 WatchSource:0}: Error finding container 87f515c5d59ad95e0bc6fe1689756f345662d5fe67277a8c87dbeeced93ca021: Status 404 returned error can't find the container with id 87f515c5d59ad95e0bc6fe1689756f345662d5fe67277a8c87dbeeced93ca021
Nov 02 23:43:21.071415 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:21.071334    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:21.542682 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:21.542646    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4" event={"ID":"28664c33-54e8-4ced-9a16-dc46980a8bf5","Type":"ContainerStarted","Data":"2f4cb97518db03d2a873358259d1722c06575dfdc7dbcff413c8805bd1add3e9"}
Nov 02 23:43:21.542858 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:21.542703    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4" event={"ID":"28664c33-54e8-4ced-9a16-dc46980a8bf5","Type":"ContainerStarted","Data":"87f515c5d59ad95e0bc6fe1689756f345662d5fe67277a8c87dbeeced93ca021"}
Nov 02 23:43:21.542858 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:21.542730    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4"
Nov 02 23:43:21.654810 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:21.654760    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4" podStartSLOduration=1.6547473959999999 podStartE2EDuration="1.654747396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:43:21.651370225 +0000 UTC m=+3041.219725014" watchObservedRunningTime="2026-11-02 23:43:21.654747396 +0000 UTC m=+3041.223102182"
Nov 02 23:43:23.525234 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:23.525198    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:24.507122 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:24.507090    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:24.519832 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:24.519800    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:24.575061 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:24.575035    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:43:25.561093 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:25.561053    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:30.163994 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:30.163940    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:31.784916 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:31.784885    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"]
Nov 02 23:43:31.785419 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:31.785159    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n" podUID="da47e811-c010-4279-aa35-82c34f9b00cb" containerName="istio-proxy" containerID="cri-o://a71f39b826cf260dd990c1451b725121870141f27a599340c8f26fb6bfdf03de" gracePeriod=30
Nov 02 23:43:32.553465 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:32.553437    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4"
Nov 02 23:43:34.737643 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:34.737611    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:36.439265 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:36.439230    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:36.843267 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:43:36.843228    4294 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda47e811_c010_4279_aa35_82c34f9b00cb.slice/crio-a71f39b826cf260dd990c1451b725121870141f27a599340c8f26fb6bfdf03de.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 23:43:37.043587 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.043557    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:43:37.131643 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.131578    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da47e811-c010-4279-aa35-82c34f9b00cb-istio-token\" (UniqueName: \"kubernetes.io/projected/da47e811-c010-4279-aa35-82c34f9b00cb-istio-token\") pod \"da47e811-c010-4279-aa35-82c34f9b00cb\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") "
Nov 02 23:43:37.131643 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.131623    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-istio-envoy\") pod \"da47e811-c010-4279-aa35-82c34f9b00cb\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") "
Nov 02 23:43:37.131867 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.131672    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da47e811-c010-4279-aa35-82c34f9b00cb-kube-api-access-xn4dl\" (UniqueName: \"kubernetes.io/projected/da47e811-c010-4279-aa35-82c34f9b00cb-kube-api-access-xn4dl\") pod \"da47e811-c010-4279-aa35-82c34f9b00cb\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") "
Nov 02 23:43:37.131867 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.131716    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-workload-certs\") pod \"da47e811-c010-4279-aa35-82c34f9b00cb\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") "
Nov 02 23:43:37.131867 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.131740    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da47e811-c010-4279-aa35-82c34f9b00cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da47e811-c010-4279-aa35-82c34f9b00cb-istio-podinfo\") pod \"da47e811-c010-4279-aa35-82c34f9b00cb\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") "
Nov 02 23:43:37.131867 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.131796    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-workload-socket\") pod \"da47e811-c010-4279-aa35-82c34f9b00cb\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") "
Nov 02 23:43:37.131867 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.131834    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-istio-data\") pod \"da47e811-c010-4279-aa35-82c34f9b00cb\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") "
Nov 02 23:43:37.132159 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.131960    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da47e811-c010-4279-aa35-82c34f9b00cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da47e811-c010-4279-aa35-82c34f9b00cb-istiod-ca-cert\") pod \"da47e811-c010-4279-aa35-82c34f9b00cb\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") "
Nov 02 23:43:37.132159 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.132041    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-workload-certs" pod "da47e811-c010-4279-aa35-82c34f9b00cb" (UID: "da47e811-c010-4279-aa35-82c34f9b00cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:43:37.132332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.132192    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-credential-socket\") pod \"da47e811-c010-4279-aa35-82c34f9b00cb\" (UID: \"da47e811-c010-4279-aa35-82c34f9b00cb\") "
Nov 02 23:43:37.132422 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.132392    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da47e811-c010-4279-aa35-82c34f9b00cb-istiod-ca-cert" pod "da47e811-c010-4279-aa35-82c34f9b00cb" (UID: "da47e811-c010-4279-aa35-82c34f9b00cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:43:37.132642 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.132560    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-workload-socket" pod "da47e811-c010-4279-aa35-82c34f9b00cb" (UID: "da47e811-c010-4279-aa35-82c34f9b00cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:43:37.132642 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.132589    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-credential-socket" pod "da47e811-c010-4279-aa35-82c34f9b00cb" (UID: "da47e811-c010-4279-aa35-82c34f9b00cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:43:37.133089 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.133000    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:43:37.133089 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.133030    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da47e811-c010-4279-aa35-82c34f9b00cb-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:43:37.133089 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.133045    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:43:37.133089 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.133049    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-istio-data" pod "da47e811-c010-4279-aa35-82c34f9b00cb" (UID: "da47e811-c010-4279-aa35-82c34f9b00cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:43:37.133089 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.133059    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:43:37.134266 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.134242    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da47e811-c010-4279-aa35-82c34f9b00cb-kube-api-access-xn4dl" pod "da47e811-c010-4279-aa35-82c34f9b00cb" (UID: "da47e811-c010-4279-aa35-82c34f9b00cb"). InnerVolumeSpecName "kube-api-access-xn4dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:43:37.134424 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.134392    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-istio-envoy" pod "da47e811-c010-4279-aa35-82c34f9b00cb" (UID: "da47e811-c010-4279-aa35-82c34f9b00cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:43:37.134528 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.134404    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da47e811-c010-4279-aa35-82c34f9b00cb-istio-token" pod "da47e811-c010-4279-aa35-82c34f9b00cb" (UID: "da47e811-c010-4279-aa35-82c34f9b00cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:43:37.134797 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.134589    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da47e811-c010-4279-aa35-82c34f9b00cb-istio-podinfo" pod "da47e811-c010-4279-aa35-82c34f9b00cb" (UID: "da47e811-c010-4279-aa35-82c34f9b00cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:43:37.234168 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.234138    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da47e811-c010-4279-aa35-82c34f9b00cb-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:43:37.234168 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.234165    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:43:37.234361 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.234179    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn4dl\" (UniqueName: \"kubernetes.io/projected/da47e811-c010-4279-aa35-82c34f9b00cb-kube-api-access-xn4dl\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:43:37.234361 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.234193    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da47e811-c010-4279-aa35-82c34f9b00cb-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:43:37.234361 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.234206    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da47e811-c010-4279-aa35-82c34f9b00cb-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:43:37.645499 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.645463    4294 generic.go:356] "Generic (PLEG): container finished" podID="da47e811-c010-4279-aa35-82c34f9b00cb" containerID="a71f39b826cf260dd990c1451b725121870141f27a599340c8f26fb6bfdf03de" exitCode=0
Nov 02 23:43:37.645969 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.645511    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"
Nov 02 23:43:37.645969 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.645550    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n" event={"ID":"da47e811-c010-4279-aa35-82c34f9b00cb","Type":"ContainerDied","Data":"a71f39b826cf260dd990c1451b725121870141f27a599340c8f26fb6bfdf03de"}
Nov 02 23:43:37.645969 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.645617    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n" event={"ID":"da47e811-c010-4279-aa35-82c34f9b00cb","Type":"ContainerDied","Data":"f0540524334e79cfe7977fe560086f9cd215c6f054b4781fc9616b85883788a3"}
Nov 02 23:43:37.645969 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.645651    4294 scope.go:122] "RemoveContainer" containerID="a71f39b826cf260dd990c1451b725121870141f27a599340c8f26fb6bfdf03de"
Nov 02 23:43:37.656904 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.656889    4294 scope.go:122] "RemoveContainer" containerID="a71f39b826cf260dd990c1451b725121870141f27a599340c8f26fb6bfdf03de"
Nov 02 23:43:37.657177 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:43:37.657160    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a71f39b826cf260dd990c1451b725121870141f27a599340c8f26fb6bfdf03de\": container with ID starting with a71f39b826cf260dd990c1451b725121870141f27a599340c8f26fb6bfdf03de not found: ID does not exist" containerID="a71f39b826cf260dd990c1451b725121870141f27a599340c8f26fb6bfdf03de"
Nov 02 23:43:37.657233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.657185    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a71f39b826cf260dd990c1451b725121870141f27a599340c8f26fb6bfdf03de"} err="failed to get container status \"a71f39b826cf260dd990c1451b725121870141f27a599340c8f26fb6bfdf03de\": rpc error: code = NotFound desc = could not find container \"a71f39b826cf260dd990c1451b725121870141f27a599340c8f26fb6bfdf03de\": container with ID starting with a71f39b826cf260dd990c1451b725121870141f27a599340c8f26fb6bfdf03de not found: ID does not exist"
Nov 02 23:43:37.804245 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.804217    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"]
Nov 02 23:43:37.851432 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:37.851400    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9cb6n"]
Nov 02 23:43:38.548737 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:38.548708    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da47e811-c010-4279-aa35-82c34f9b00cb" path="/var/lib/kubelet/pods/da47e811-c010-4279-aa35-82c34f9b00cb/volumes"
Nov 02 23:43:38.799023 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:38.798919    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:39.019667 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:39.019636    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"]
Nov 02 23:43:39.019857 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:39.019837    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c" podUID="2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" containerName="istio-proxy" containerID="cri-o://577421ce91774a38149c9e439ced871fe093be2e26c17776e876023d5be45146" gracePeriod=30
Nov 02 23:43:39.332996 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:39.332953    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:41.753136 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:43:41.753098    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:43:43.553916 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:43.553875    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 23:43:43.554410 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:43.553931    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c" podUID="2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 23:43:43.831028 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:43.830929    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:44.290425 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.290399    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:43:44.421224 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.421136    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-credential-socket\") pod \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") "
Nov 02 23:43:44.421224 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.421213    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-podinfo\") pod \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") "
Nov 02 23:43:44.421448 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.421425    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-workload-certs\") pod \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") "
Nov 02 23:43:44.421503 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.421492    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-envoy\") pod \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") "
Nov 02 23:43:44.421567 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.421550    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-credential-socket" pod "2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" (UID: "2ec3e6ec-079a-485e-a7f3-3f830c7b89ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:43:44.421627 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.421579    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-token\" (UniqueName: \"kubernetes.io/projected/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-token\") pod \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") "
Nov 02 23:43:44.421627 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.421610    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-workload-socket\") pod \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") "
Nov 02 23:43:44.421751 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.421665    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-workload-certs" pod "2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" (UID: "2ec3e6ec-079a-485e-a7f3-3f830c7b89ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:43:44.421751 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.421673    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-data\") pod \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") "
Nov 02 23:43:44.421751 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.421720    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-kube-api-access-ghmrr\" (UniqueName: \"kubernetes.io/projected/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-kube-api-access-ghmrr\") pod \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") "
Nov 02 23:43:44.421911 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.421786    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istiod-ca-cert\") pod \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\" (UID: \"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec\") "
Nov 02 23:43:44.421911 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.421877    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-data" pod "2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" (UID: "2ec3e6ec-079a-485e-a7f3-3f830c7b89ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:43:44.421911 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.421890    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-workload-socket" pod "2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" (UID: "2ec3e6ec-079a-485e-a7f3-3f830c7b89ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:43:44.422381 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.422355    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:43:44.422515 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.422385    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:43:44.422515 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.422397    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istiod-ca-cert" pod "2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" (UID: "2ec3e6ec-079a-485e-a7f3-3f830c7b89ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:43:44.422515 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.422402    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:43:44.422515 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.422449    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:43:44.423749 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.423718    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-podinfo" pod "2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" (UID: "2ec3e6ec-079a-485e-a7f3-3f830c7b89ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:43:44.423867 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.423837    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-token" pod "2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" (UID: "2ec3e6ec-079a-485e-a7f3-3f830c7b89ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:43:44.424161 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.424135    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-envoy" pod "2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" (UID: "2ec3e6ec-079a-485e-a7f3-3f830c7b89ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:43:44.424259 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.424139    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-kube-api-access-ghmrr" pod "2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" (UID: "2ec3e6ec-079a-485e-a7f3-3f830c7b89ec"). InnerVolumeSpecName "kube-api-access-ghmrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:43:44.524116 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.524078    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:43:44.524116 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.524116    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:43:44.524316 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.524131    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:43:44.524316 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.524145    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghmrr\" (UniqueName: \"kubernetes.io/projected/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-kube-api-access-ghmrr\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:43:44.524316 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.524160    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:43:44.690735 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.690653    4294 generic.go:356] "Generic (PLEG): container finished" podID="2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" containerID="577421ce91774a38149c9e439ced871fe093be2e26c17776e876023d5be45146" exitCode=0
Nov 02 23:43:44.690735 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.690722    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"
Nov 02 23:43:44.690735 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.690727    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c" event={"ID":"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec","Type":"ContainerDied","Data":"577421ce91774a38149c9e439ced871fe093be2e26c17776e876023d5be45146"}
Nov 02 23:43:44.691246 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.690780    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c" event={"ID":"2ec3e6ec-079a-485e-a7f3-3f830c7b89ec","Type":"ContainerDied","Data":"e506a6e7cb651e8bd7ccb5c24019c697e468f5f5d4eb555c6d9faf5e2fc154fc"}
Nov 02 23:43:44.691246 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.690810    4294 scope.go:122] "RemoveContainer" containerID="577421ce91774a38149c9e439ced871fe093be2e26c17776e876023d5be45146"
Nov 02 23:43:44.700707 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.700681    4294 scope.go:122] "RemoveContainer" containerID="577421ce91774a38149c9e439ced871fe093be2e26c17776e876023d5be45146"
Nov 02 23:43:44.700925 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:43:44.700903    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"577421ce91774a38149c9e439ced871fe093be2e26c17776e876023d5be45146\": container with ID starting with 577421ce91774a38149c9e439ced871fe093be2e26c17776e876023d5be45146 not found: ID does not exist" containerID="577421ce91774a38149c9e439ced871fe093be2e26c17776e876023d5be45146"
Nov 02 23:43:44.701004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.700935    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577421ce91774a38149c9e439ced871fe093be2e26c17776e876023d5be45146"} err="failed to get container status \"577421ce91774a38149c9e439ced871fe093be2e26c17776e876023d5be45146\": rpc error: code = NotFound desc = could not find container \"577421ce91774a38149c9e439ced871fe093be2e26c17776e876023d5be45146\": container with ID starting with 577421ce91774a38149c9e439ced871fe093be2e26c17776e876023d5be45146 not found: ID does not exist"
Nov 02 23:43:44.857866 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.857835    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"]
Nov 02 23:43:44.904887 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:44.904856    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-hhv5c"]
Nov 02 23:43:46.548461 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:46.548430    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" path="/var/lib/kubelet/pods/2ec3e6ec-079a-485e-a7f3-3f830c7b89ec/volumes"
Nov 02 23:43:48.350598 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:48.350563    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:49.956036 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:49.956005    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:52.834610 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:52.834572    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:55.822674 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:55.822635    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:43:57.329881 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:43:57.329835    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:44:01.877105 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:44:01.877075    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:44:06.438130 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:44:06.438091    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:44:11.458813 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:44:11.458781    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:44:11.756674 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:44:11.756616    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:44:16.159892 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:44:16.159853    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:44:21.000499 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:44:21.000470    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:44:21.168442 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:44:21.168408    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:44:25.984245 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:44:25.984162    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:44:30.642372 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:44:30.642338    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:44:32.708584 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:44:32.708548    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:44:35.182788 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:44:35.182752    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:44:39.177337 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:44:39.177305    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:44:39.713554 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:44:39.713520    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:44:41.751957 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:44:41.751919    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:44:44.234592 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:44:44.234562    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:44:48.774920 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:44:48.774883    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:44:53.350454 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:44:53.350420    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:44:57.870861 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:44:57.870827    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:45:00.190466 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.190437    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55"]
Nov 02 23:45:00.191120 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.191099    4294 cpu_manager.go:414] "containerMap: removing container" podUID="da47e811-c010-4279-aa35-82c34f9b00cb" containerName="istio-proxy"
Nov 02 23:45:00.191120 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.191117    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da47e811-c010-4279-aa35-82c34f9b00cb" containerName="istio-proxy"
Nov 02 23:45:00.191295 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.191146    4294 cpu_manager.go:414] "containerMap: removing container" podUID="2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" containerName="istio-proxy"
Nov 02 23:45:00.191295 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.191151    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" containerName="istio-proxy"
Nov 02 23:45:00.191295 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.191248    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec3e6ec-079a-485e-a7f3-3f830c7b89ec" containerName="istio-proxy"
Nov 02 23:45:00.191295 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.191263    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="da47e811-c010-4279-aa35-82c34f9b00cb" containerName="istio-proxy"
Nov 02 23:45:00.194699 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.194684    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc"]
Nov 02 23:45:00.194829 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.194810    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55"
Nov 02 23:45:00.198860 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.198839    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc"
Nov 02 23:45:00.208361 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.208333    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/156f1da2-39a1-45bc-a8c0-248a57b78e10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55\" (UID: \"156f1da2-39a1-45bc-a8c0-248a57b78e10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55"
Nov 02 23:45:00.208494 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.208388    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f1b525-8ff0-448c-aba4-333082f26bf9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc\" (UID: \"28f1b525-8ff0-448c-aba4-333082f26bf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc"
Nov 02 23:45:00.208494 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.208479    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64rsm\" (UniqueName: \"kubernetes.io/projected/156f1da2-39a1-45bc-a8c0-248a57b78e10-kube-api-access-64rsm\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55\" (UID: \"156f1da2-39a1-45bc-a8c0-248a57b78e10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55"
Nov 02 23:45:00.208624 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.208516    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt8kq\" (UniqueName: \"kubernetes.io/projected/28f1b525-8ff0-448c-aba4-333082f26bf9-kube-api-access-rt8kq\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc\" (UID: \"28f1b525-8ff0-448c-aba4-333082f26bf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc"
Nov 02 23:45:00.209918 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.209899    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf"]
Nov 02 23:45:00.214065 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.214049    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf"
Nov 02 23:45:00.237952 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.237930    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc"]
Nov 02 23:45:00.241797 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.241782    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc"
Nov 02 23:45:00.309795 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.309756    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe05a3f-2410-4cbc-adab-33b1fa3625e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc\" (UID: \"bbe05a3f-2410-4cbc-adab-33b1fa3625e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc"
Nov 02 23:45:00.310439 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.309828    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ed09312-d986-44b0-8f95-92349baa7d36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf\" (UID: \"8ed09312-d986-44b0-8f95-92349baa7d36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf"
Nov 02 23:45:00.310439 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.309973    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/156f1da2-39a1-45bc-a8c0-248a57b78e10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55\" (UID: \"156f1da2-39a1-45bc-a8c0-248a57b78e10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55"
Nov 02 23:45:00.310439 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.310055    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f1b525-8ff0-448c-aba4-333082f26bf9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc\" (UID: \"28f1b525-8ff0-448c-aba4-333082f26bf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc"
Nov 02 23:45:00.310439 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.310113    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64rsm\" (UniqueName: \"kubernetes.io/projected/156f1da2-39a1-45bc-a8c0-248a57b78e10-kube-api-access-64rsm\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55\" (UID: \"156f1da2-39a1-45bc-a8c0-248a57b78e10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55"
Nov 02 23:45:00.310439 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.310143    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt8kq\" (UniqueName: \"kubernetes.io/projected/28f1b525-8ff0-448c-aba4-333082f26bf9-kube-api-access-rt8kq\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc\" (UID: \"28f1b525-8ff0-448c-aba4-333082f26bf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc"
Nov 02 23:45:00.310439 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.310215    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r57tx\" (UniqueName: \"kubernetes.io/projected/bbe05a3f-2410-4cbc-adab-33b1fa3625e3-kube-api-access-r57tx\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc\" (UID: \"bbe05a3f-2410-4cbc-adab-33b1fa3625e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc"
Nov 02 23:45:00.310439 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.310249    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grvcc\" (UniqueName: \"kubernetes.io/projected/8ed09312-d986-44b0-8f95-92349baa7d36-kube-api-access-grvcc\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf\" (UID: \"8ed09312-d986-44b0-8f95-92349baa7d36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf"
Nov 02 23:45:00.311079 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.311055    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/156f1da2-39a1-45bc-a8c0-248a57b78e10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55\" (UID: \"156f1da2-39a1-45bc-a8c0-248a57b78e10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55"
Nov 02 23:45:00.311691 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.311667    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f1b525-8ff0-448c-aba4-333082f26bf9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc\" (UID: \"28f1b525-8ff0-448c-aba4-333082f26bf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc"
Nov 02 23:45:00.323294 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.323270    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\""
Nov 02 23:45:00.356313 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.352860    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc"]
Nov 02 23:45:00.359547 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.359434    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf"]
Nov 02 23:45:00.363102 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.363080    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc"]
Nov 02 23:45:00.369054 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.369030    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64rsm\" (UniqueName: \"kubernetes.io/projected/156f1da2-39a1-45bc-a8c0-248a57b78e10-kube-api-access-64rsm\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55\" (UID: \"156f1da2-39a1-45bc-a8c0-248a57b78e10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55"
Nov 02 23:45:00.369545 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.369518    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt8kq\" (UniqueName: \"kubernetes.io/projected/28f1b525-8ff0-448c-aba4-333082f26bf9-kube-api-access-rt8kq\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc\" (UID: \"28f1b525-8ff0-448c-aba4-333082f26bf9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc"
Nov 02 23:45:00.377297 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.377271    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55"]
Nov 02 23:45:00.411191 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.411161    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r57tx\" (UniqueName: \"kubernetes.io/projected/bbe05a3f-2410-4cbc-adab-33b1fa3625e3-kube-api-access-r57tx\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc\" (UID: \"bbe05a3f-2410-4cbc-adab-33b1fa3625e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc"
Nov 02 23:45:00.411191 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.411201    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grvcc\" (UniqueName: \"kubernetes.io/projected/8ed09312-d986-44b0-8f95-92349baa7d36-kube-api-access-grvcc\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf\" (UID: \"8ed09312-d986-44b0-8f95-92349baa7d36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf"
Nov 02 23:45:00.411412 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.411229    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe05a3f-2410-4cbc-adab-33b1fa3625e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc\" (UID: \"bbe05a3f-2410-4cbc-adab-33b1fa3625e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc"
Nov 02 23:45:00.411412 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.411274    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ed09312-d986-44b0-8f95-92349baa7d36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf\" (UID: \"8ed09312-d986-44b0-8f95-92349baa7d36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf"
Nov 02 23:45:00.412060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.412035    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ed09312-d986-44b0-8f95-92349baa7d36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf\" (UID: \"8ed09312-d986-44b0-8f95-92349baa7d36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf"
Nov 02 23:45:00.412161 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.412066    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe05a3f-2410-4cbc-adab-33b1fa3625e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc\" (UID: \"bbe05a3f-2410-4cbc-adab-33b1fa3625e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc"
Nov 02 23:45:00.465270 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.465205    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r57tx\" (UniqueName: \"kubernetes.io/projected/bbe05a3f-2410-4cbc-adab-33b1fa3625e3-kube-api-access-r57tx\") pod \"maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc\" (UID: \"bbe05a3f-2410-4cbc-adab-33b1fa3625e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc"
Nov 02 23:45:00.465372 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.465293    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grvcc\" (UniqueName: \"kubernetes.io/projected/8ed09312-d986-44b0-8f95-92349baa7d36-kube-api-access-grvcc\") pod \"maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf\" (UID: \"8ed09312-d986-44b0-8f95-92349baa7d36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf"
Nov 02 23:45:00.511730 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.511699    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55"
Nov 02 23:45:00.518128 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.518104    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc"
Nov 02 23:45:00.527065 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.527047    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf"
Nov 02 23:45:00.555198 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.555173    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc"
Nov 02 23:45:00.884398 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:45:00.884366    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156f1da2_39a1_45bc_a8c0_248a57b78e10.slice/crio-bd4eb146eccc3aaa57a0db2ba4f1e9df21de17e2513bb6a7d3767ab6b06810f6 WatchSource:0}: Error finding container bd4eb146eccc3aaa57a0db2ba4f1e9df21de17e2513bb6a7d3767ab6b06810f6: Status 404 returned error can't find the container with id bd4eb146eccc3aaa57a0db2ba4f1e9df21de17e2513bb6a7d3767ab6b06810f6
Nov 02 23:45:00.886566 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.886545    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55"]
Nov 02 23:45:00.931091 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:45:00.931068    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f1b525_8ff0_448c_aba4_333082f26bf9.slice/crio-0a63d1863544ba354b769e1818ae25000b9d5096ef1a65089ec10c4453ac3590 WatchSource:0}: Error finding container 0a63d1863544ba354b769e1818ae25000b9d5096ef1a65089ec10c4453ac3590: Status 404 returned error can't find the container with id 0a63d1863544ba354b769e1818ae25000b9d5096ef1a65089ec10c4453ac3590
Nov 02 23:45:00.931186 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.931169    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc"]
Nov 02 23:45:00.932768 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:45:00.932747    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed09312_d986_44b0_8f95_92349baa7d36.slice/crio-b83b55329232a709732c388a73ddd7f798364d8482e4114e6c55cb3262725d84 WatchSource:0}: Error finding container b83b55329232a709732c388a73ddd7f798364d8482e4114e6c55cb3262725d84: Status 404 returned error can't find the container with id b83b55329232a709732c388a73ddd7f798364d8482e4114e6c55cb3262725d84
Nov 02 23:45:00.935587 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.935570    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf"]
Nov 02 23:45:00.965224 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:00.965205    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc"]
Nov 02 23:45:01.055753 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:45:01.055718    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe05a3f_2410_4cbc_adab_33b1fa3625e3.slice/crio-339db66f830efc3876f9518477a47e5d38a54418fffc81c76995e213efac5534 WatchSource:0}: Error finding container 339db66f830efc3876f9518477a47e5d38a54418fffc81c76995e213efac5534: Status 404 returned error can't find the container with id 339db66f830efc3876f9518477a47e5d38a54418fffc81c76995e213efac5534
Nov 02 23:45:01.196190 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:01.196157    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55" event={"ID":"156f1da2-39a1-45bc-a8c0-248a57b78e10","Type":"ContainerStarted","Data":"bd4eb146eccc3aaa57a0db2ba4f1e9df21de17e2513bb6a7d3767ab6b06810f6"}
Nov 02 23:45:01.197535 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:01.197508    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf" event={"ID":"8ed09312-d986-44b0-8f95-92349baa7d36","Type":"ContainerStarted","Data":"b83b55329232a709732c388a73ddd7f798364d8482e4114e6c55cb3262725d84"}
Nov 02 23:45:01.198822 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:01.198797    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc" event={"ID":"bbe05a3f-2410-4cbc-adab-33b1fa3625e3","Type":"ContainerStarted","Data":"339db66f830efc3876f9518477a47e5d38a54418fffc81c76995e213efac5534"}
Nov 02 23:45:01.200724 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:01.200701    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc" event={"ID":"28f1b525-8ff0-448c-aba4-333082f26bf9","Type":"ContainerStarted","Data":"0a63d1863544ba354b769e1818ae25000b9d5096ef1a65089ec10c4453ac3590"}
Nov 02 23:45:02.209037 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:02.209004    4294 generic.go:356] "Generic (PLEG): container finished" podID="156f1da2-39a1-45bc-a8c0-248a57b78e10" containerID="18a4ce56f631ee1bc4d63430606b94fdecc1db6278bf962141edc21a576c7e0b" exitCode=60
Nov 02 23:45:02.209431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:02.209085    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55" event={"ID":"156f1da2-39a1-45bc-a8c0-248a57b78e10","Type":"ContainerDied","Data":"18a4ce56f631ee1bc4d63430606b94fdecc1db6278bf962141edc21a576c7e0b"}
Nov 02 23:45:02.209431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:02.209347    4294 scope.go:122] "RemoveContainer" containerID="18a4ce56f631ee1bc4d63430606b94fdecc1db6278bf962141edc21a576c7e0b"
Nov 02 23:45:02.210711 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:02.210688    4294 generic.go:356] "Generic (PLEG): container finished" podID="8ed09312-d986-44b0-8f95-92349baa7d36" containerID="b2533c8d73c71d5008f92e5a7451dd9f424528a74ea7bd0c334bc66271ccdf98" exitCode=60
Nov 02 23:45:02.210822 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:02.210779    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf" event={"ID":"8ed09312-d986-44b0-8f95-92349baa7d36","Type":"ContainerDied","Data":"b2533c8d73c71d5008f92e5a7451dd9f424528a74ea7bd0c334bc66271ccdf98"}
Nov 02 23:45:02.211037 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:02.211021    4294 scope.go:122] "RemoveContainer" containerID="b2533c8d73c71d5008f92e5a7451dd9f424528a74ea7bd0c334bc66271ccdf98"
Nov 02 23:45:02.212498 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:02.212447    4294 generic.go:356] "Generic (PLEG): container finished" podID="bbe05a3f-2410-4cbc-adab-33b1fa3625e3" containerID="7ecd12bdd596aacb8bb2bc938ddb80ccfd018991f02c43aa00f4401512e8b6bf" exitCode=60
Nov 02 23:45:02.212570 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:02.212506    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc" event={"ID":"bbe05a3f-2410-4cbc-adab-33b1fa3625e3","Type":"ContainerDied","Data":"7ecd12bdd596aacb8bb2bc938ddb80ccfd018991f02c43aa00f4401512e8b6bf"}
Nov 02 23:45:02.212765 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:02.212749    4294 scope.go:122] "RemoveContainer" containerID="7ecd12bdd596aacb8bb2bc938ddb80ccfd018991f02c43aa00f4401512e8b6bf"
Nov 02 23:45:02.214261 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:02.214244    4294 generic.go:356] "Generic (PLEG): container finished" podID="28f1b525-8ff0-448c-aba4-333082f26bf9" containerID="e048b9d005afe992eb5669fa4ab2ed130d1686dbc42aaedb48d7449edc4cad93" exitCode=60
Nov 02 23:45:02.214335 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:02.214322    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc" event={"ID":"28f1b525-8ff0-448c-aba4-333082f26bf9","Type":"ContainerDied","Data":"e048b9d005afe992eb5669fa4ab2ed130d1686dbc42aaedb48d7449edc4cad93"}
Nov 02 23:45:02.214539 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:02.214525    4294 scope.go:122] "RemoveContainer" containerID="e048b9d005afe992eb5669fa4ab2ed130d1686dbc42aaedb48d7449edc4cad93"
Nov 02 23:45:02.450033 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:02.449932    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:45:03.222788 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:03.222757    4294 generic.go:356] "Generic (PLEG): container finished" podID="8ed09312-d986-44b0-8f95-92349baa7d36" containerID="4273ec028c4819ccccc1841bb944b4cc6ea0bab3ec5d296105caa7fc85b2aa21" exitCode=60
Nov 02 23:45:03.223292 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:03.222842    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf" event={"ID":"8ed09312-d986-44b0-8f95-92349baa7d36","Type":"ContainerDied","Data":"4273ec028c4819ccccc1841bb944b4cc6ea0bab3ec5d296105caa7fc85b2aa21"}
Nov 02 23:45:03.223292 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:03.222904    4294 scope.go:122] "RemoveContainer" containerID="b2533c8d73c71d5008f92e5a7451dd9f424528a74ea7bd0c334bc66271ccdf98"
Nov 02 23:45:03.223292 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:03.223189    4294 scope.go:122] "RemoveContainer" containerID="4273ec028c4819ccccc1841bb944b4cc6ea0bab3ec5d296105caa7fc85b2aa21"
Nov 02 23:45:03.223579 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:45:03.223557    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf_odh-ai-gateway-infra(8ed09312-d986-44b0-8f95-92349baa7d36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf" podUID="8ed09312-d986-44b0-8f95-92349baa7d36"
Nov 02 23:45:03.225004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:03.224962    4294 generic.go:356] "Generic (PLEG): container finished" podID="bbe05a3f-2410-4cbc-adab-33b1fa3625e3" containerID="582298253c350b95b0021e424959cfd55e6f7e28c5ee5c30168da9c3633d4686" exitCode=60
Nov 02 23:45:03.225118 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:03.225022    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc" event={"ID":"bbe05a3f-2410-4cbc-adab-33b1fa3625e3","Type":"ContainerDied","Data":"582298253c350b95b0021e424959cfd55e6f7e28c5ee5c30168da9c3633d4686"}
Nov 02 23:45:03.225348 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:03.225331    4294 scope.go:122] "RemoveContainer" containerID="582298253c350b95b0021e424959cfd55e6f7e28c5ee5c30168da9c3633d4686"
Nov 02 23:45:03.225641 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:45:03.225623    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc_odh-ai-gateway-infra(bbe05a3f-2410-4cbc-adab-33b1fa3625e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc" podUID="bbe05a3f-2410-4cbc-adab-33b1fa3625e3"
Nov 02 23:45:03.227035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:03.227006    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc" event={"ID":"28f1b525-8ff0-448c-aba4-333082f26bf9","Type":"ContainerStarted","Data":"eb67d135acbff90013e3d6085042facc786d46158b017ae4891a45d1ae603994"}
Nov 02 23:45:03.228910 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:03.228890    4294 generic.go:356] "Generic (PLEG): container finished" podID="156f1da2-39a1-45bc-a8c0-248a57b78e10" containerID="d631c0289a263fec8b1d7474431737a5388a4554289ffd3053e0ef4ad089a5c6" exitCode=60
Nov 02 23:45:03.229041 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:03.228957    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55" event={"ID":"156f1da2-39a1-45bc-a8c0-248a57b78e10","Type":"ContainerDied","Data":"d631c0289a263fec8b1d7474431737a5388a4554289ffd3053e0ef4ad089a5c6"}
Nov 02 23:45:03.229217 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:03.229200    4294 scope.go:122] "RemoveContainer" containerID="d631c0289a263fec8b1d7474431737a5388a4554289ffd3053e0ef4ad089a5c6"
Nov 02 23:45:03.229462 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:45:03.229438    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55_odh-ai-gateway-infra(156f1da2-39a1-45bc-a8c0-248a57b78e10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55" podUID="156f1da2-39a1-45bc-a8c0-248a57b78e10"
Nov 02 23:45:03.412891 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:03.412869    4294 scope.go:122] "RemoveContainer" containerID="7ecd12bdd596aacb8bb2bc938ddb80ccfd018991f02c43aa00f4401512e8b6bf"
Nov 02 23:45:03.425873 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:03.425849    4294 scope.go:122] "RemoveContainer" containerID="18a4ce56f631ee1bc4d63430606b94fdecc1db6278bf962141edc21a576c7e0b"
Nov 02 23:45:03.546516 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:03.546459    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc" podStartSLOduration=3.546444705 podStartE2EDuration="3.546444705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:03.545637707 +0000 UTC m=+3143.113992495" watchObservedRunningTime="2026-11-02 23:45:03.546444705 +0000 UTC m=+3143.114799491"
Nov 02 23:45:04.236654 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:04.236622    4294 scope.go:122] "RemoveContainer" containerID="582298253c350b95b0021e424959cfd55e6f7e28c5ee5c30168da9c3633d4686"
Nov 02 23:45:04.237078 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:45:04.236917    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc_odh-ai-gateway-infra(bbe05a3f-2410-4cbc-adab-33b1fa3625e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc" podUID="bbe05a3f-2410-4cbc-adab-33b1fa3625e3"
Nov 02 23:45:04.238327 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:04.238299    4294 generic.go:356] "Generic (PLEG): container finished" podID="28f1b525-8ff0-448c-aba4-333082f26bf9" containerID="eb67d135acbff90013e3d6085042facc786d46158b017ae4891a45d1ae603994" exitCode=60
Nov 02 23:45:04.238584 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:04.238495    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc" event={"ID":"28f1b525-8ff0-448c-aba4-333082f26bf9","Type":"ContainerDied","Data":"eb67d135acbff90013e3d6085042facc786d46158b017ae4891a45d1ae603994"}
Nov 02 23:45:04.238584 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:04.238575    4294 scope.go:122] "RemoveContainer" containerID="e048b9d005afe992eb5669fa4ab2ed130d1686dbc42aaedb48d7449edc4cad93"
Nov 02 23:45:04.238718 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:04.238630    4294 scope.go:122] "RemoveContainer" containerID="eb67d135acbff90013e3d6085042facc786d46158b017ae4891a45d1ae603994"
Nov 02 23:45:04.238922 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:45:04.238902    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc_odh-ai-gateway-infra(28f1b525-8ff0-448c-aba4-333082f26bf9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc" podUID="28f1b525-8ff0-448c-aba4-333082f26bf9"
Nov 02 23:45:04.240430 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:04.240406    4294 scope.go:122] "RemoveContainer" containerID="d631c0289a263fec8b1d7474431737a5388a4554289ffd3053e0ef4ad089a5c6"
Nov 02 23:45:04.240638 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:45:04.240615    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55_odh-ai-gateway-infra(156f1da2-39a1-45bc-a8c0-248a57b78e10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55" podUID="156f1da2-39a1-45bc-a8c0-248a57b78e10"
Nov 02 23:45:04.241935 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:04.241916    4294 scope.go:122] "RemoveContainer" containerID="4273ec028c4819ccccc1841bb944b4cc6ea0bab3ec5d296105caa7fc85b2aa21"
Nov 02 23:45:04.242129 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:45:04.242112    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf_odh-ai-gateway-infra(8ed09312-d986-44b0-8f95-92349baa7d36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf" podUID="8ed09312-d986-44b0-8f95-92349baa7d36"
Nov 02 23:45:05.250199 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:05.250171    4294 scope.go:122] "RemoveContainer" containerID="eb67d135acbff90013e3d6085042facc786d46158b017ae4891a45d1ae603994"
Nov 02 23:45:05.250583 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:45:05.250444    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc_odh-ai-gateway-infra(28f1b525-8ff0-448c-aba4-333082f26bf9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc" podUID="28f1b525-8ff0-448c-aba4-333082f26bf9"
Nov 02 23:45:07.018127 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:07.018095    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:45:11.752026 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:45:11.751959    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:45:13.864889 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:13.864852    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:45:14.872374 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:14.872344    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:45:15.541437 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:15.541402    4294 scope.go:122] "RemoveContainer" containerID="d631c0289a263fec8b1d7474431737a5388a4554289ffd3053e0ef4ad089a5c6"
Nov 02 23:45:15.541614 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:15.541447    4294 scope.go:122] "RemoveContainer" containerID="4273ec028c4819ccccc1841bb944b4cc6ea0bab3ec5d296105caa7fc85b2aa21"
Nov 02 23:45:15.541614 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:15.541515    4294 scope.go:122] "RemoveContainer" containerID="eb67d135acbff90013e3d6085042facc786d46158b017ae4891a45d1ae603994"
Nov 02 23:45:16.321604 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:16.321568    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55" event={"ID":"156f1da2-39a1-45bc-a8c0-248a57b78e10","Type":"ContainerStarted","Data":"6db813560369100841cce694fff6269dd45973e8e696ce0a09b390ae7f3c3d49"}
Nov 02 23:45:16.323484 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:16.323457    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf" event={"ID":"8ed09312-d986-44b0-8f95-92349baa7d36","Type":"ContainerStarted","Data":"1d3db72258cf290976723e9b07c95fe92e1dc20fa35adb450ef5937ae2fc636f"}
Nov 02 23:45:16.325180 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:16.325158    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc" event={"ID":"28f1b525-8ff0-448c-aba4-333082f26bf9","Type":"ContainerStarted","Data":"7d256df3f110779ff2de5571a6ac89272b2260557c01f97ba3a070335276bc77"}
Nov 02 23:45:16.391021 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:16.390968    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:45:16.430973 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:16.430919    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55" podStartSLOduration=16.430902332 podStartE2EDuration="16.430902332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:16.429798123 +0000 UTC m=+3155.998152910" watchObservedRunningTime="2026-11-02 23:45:16.430902332 +0000 UTC m=+3155.999257168"
Nov 02 23:45:16.539192 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:16.539140    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf" podStartSLOduration=16.539127107 podStartE2EDuration="16.539127107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:16.537822667 +0000 UTC m=+3156.106177454" watchObservedRunningTime="2026-11-02 23:45:16.539127107 +0000 UTC m=+3156.107481894"
Nov 02 23:45:16.541409 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:16.541390    4294 scope.go:122] "RemoveContainer" containerID="582298253c350b95b0021e424959cfd55e6f7e28c5ee5c30168da9c3633d4686"
Nov 02 23:45:17.333210 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:17.333179    4294 generic.go:356] "Generic (PLEG): container finished" podID="156f1da2-39a1-45bc-a8c0-248a57b78e10" containerID="6db813560369100841cce694fff6269dd45973e8e696ce0a09b390ae7f3c3d49" exitCode=60
Nov 02 23:45:17.333654 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:17.333257    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55" event={"ID":"156f1da2-39a1-45bc-a8c0-248a57b78e10","Type":"ContainerDied","Data":"6db813560369100841cce694fff6269dd45973e8e696ce0a09b390ae7f3c3d49"}
Nov 02 23:45:17.333654 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:17.333319    4294 scope.go:122] "RemoveContainer" containerID="d631c0289a263fec8b1d7474431737a5388a4554289ffd3053e0ef4ad089a5c6"
Nov 02 23:45:17.333771 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:17.333660    4294 scope.go:122] "RemoveContainer" containerID="6db813560369100841cce694fff6269dd45973e8e696ce0a09b390ae7f3c3d49"
Nov 02 23:45:17.334011 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:45:17.333975    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55_odh-ai-gateway-infra(156f1da2-39a1-45bc-a8c0-248a57b78e10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55" podUID="156f1da2-39a1-45bc-a8c0-248a57b78e10"
Nov 02 23:45:17.335254 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:17.335227    4294 generic.go:356] "Generic (PLEG): container finished" podID="8ed09312-d986-44b0-8f95-92349baa7d36" containerID="1d3db72258cf290976723e9b07c95fe92e1dc20fa35adb450ef5937ae2fc636f" exitCode=60
Nov 02 23:45:17.335366 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:17.335289    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf" event={"ID":"8ed09312-d986-44b0-8f95-92349baa7d36","Type":"ContainerDied","Data":"1d3db72258cf290976723e9b07c95fe92e1dc20fa35adb450ef5937ae2fc636f"}
Nov 02 23:45:17.335640 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:17.335622    4294 scope.go:122] "RemoveContainer" containerID="1d3db72258cf290976723e9b07c95fe92e1dc20fa35adb450ef5937ae2fc636f"
Nov 02 23:45:17.335903 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:45:17.335878    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf_odh-ai-gateway-infra(8ed09312-d986-44b0-8f95-92349baa7d36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf" podUID="8ed09312-d986-44b0-8f95-92349baa7d36"
Nov 02 23:45:17.337317 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:17.337290    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc" event={"ID":"bbe05a3f-2410-4cbc-adab-33b1fa3625e3","Type":"ContainerStarted","Data":"816c3b349bbf95ae45d51fb1043451b62f45427e260dac0195ec3fa32770cc46"}
Nov 02 23:45:17.339273 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:17.339250    4294 generic.go:356] "Generic (PLEG): container finished" podID="28f1b525-8ff0-448c-aba4-333082f26bf9" containerID="7d256df3f110779ff2de5571a6ac89272b2260557c01f97ba3a070335276bc77" exitCode=60
Nov 02 23:45:17.339357 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:17.339282    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc" event={"ID":"28f1b525-8ff0-448c-aba4-333082f26bf9","Type":"ContainerDied","Data":"7d256df3f110779ff2de5571a6ac89272b2260557c01f97ba3a070335276bc77"}
Nov 02 23:45:17.339615 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:17.339598    4294 scope.go:122] "RemoveContainer" containerID="7d256df3f110779ff2de5571a6ac89272b2260557c01f97ba3a070335276bc77"
Nov 02 23:45:17.339873 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:45:17.339851    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc_odh-ai-gateway-infra(28f1b525-8ff0-448c-aba4-333082f26bf9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc" podUID="28f1b525-8ff0-448c-aba4-333082f26bf9"
Nov 02 23:45:17.346672 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:17.346653    4294 scope.go:122] "RemoveContainer" containerID="4273ec028c4819ccccc1841bb944b4cc6ea0bab3ec5d296105caa7fc85b2aa21"
Nov 02 23:45:17.360734 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:17.360718    4294 scope.go:122] "RemoveContainer" containerID="eb67d135acbff90013e3d6085042facc786d46158b017ae4891a45d1ae603994"
Nov 02 23:45:17.440964 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:17.440918    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55"]
Nov 02 23:45:17.547994 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:17.547951    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf"]
Nov 02 23:45:17.596448 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:17.596359    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc" podStartSLOduration=17.59634654 podStartE2EDuration="17.59634654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:17.593833382 +0000 UTC m=+3157.162188151" watchObservedRunningTime="2026-11-02 23:45:17.59634654 +0000 UTC m=+3157.164701326"
Nov 02 23:45:17.662287 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:17.662260    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc"]
Nov 02 23:45:18.351041 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.351012    4294 generic.go:356] "Generic (PLEG): container finished" podID="bbe05a3f-2410-4cbc-adab-33b1fa3625e3" containerID="816c3b349bbf95ae45d51fb1043451b62f45427e260dac0195ec3fa32770cc46" exitCode=60
Nov 02 23:45:18.351453 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.351070    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc" event={"ID":"bbe05a3f-2410-4cbc-adab-33b1fa3625e3","Type":"ContainerDied","Data":"816c3b349bbf95ae45d51fb1043451b62f45427e260dac0195ec3fa32770cc46"}
Nov 02 23:45:18.351453 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.351124    4294 scope.go:122] "RemoveContainer" containerID="582298253c350b95b0021e424959cfd55e6f7e28c5ee5c30168da9c3633d4686"
Nov 02 23:45:18.351453 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.351434    4294 scope.go:122] "RemoveContainer" containerID="816c3b349bbf95ae45d51fb1043451b62f45427e260dac0195ec3fa32770cc46"
Nov 02 23:45:18.351760 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:45:18.351734    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc_odh-ai-gateway-infra(bbe05a3f-2410-4cbc-adab-33b1fa3625e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc" podUID="bbe05a3f-2410-4cbc-adab-33b1fa3625e3"
Nov 02 23:45:18.508680 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.508653    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55"
Nov 02 23:45:18.592415 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.592390    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc"
Nov 02 23:45:18.596582 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.596562    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf"
Nov 02 23:45:18.605071 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.605017    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc"]
Nov 02 23:45:18.642176 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.642153    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/156f1da2-39a1-45bc-a8c0-248a57b78e10-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/156f1da2-39a1-45bc-a8c0-248a57b78e10-maas-api-service-ca\") pod \"156f1da2-39a1-45bc-a8c0-248a57b78e10\" (UID: \"156f1da2-39a1-45bc-a8c0-248a57b78e10\") "
Nov 02 23:45:18.642275 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.642209    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156f1da2-39a1-45bc-a8c0-248a57b78e10-kube-api-access-64rsm\" (UniqueName: \"kubernetes.io/projected/156f1da2-39a1-45bc-a8c0-248a57b78e10-kube-api-access-64rsm\") pod \"156f1da2-39a1-45bc-a8c0-248a57b78e10\" (UID: \"156f1da2-39a1-45bc-a8c0-248a57b78e10\") "
Nov 02 23:45:18.642473 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.642454    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156f1da2-39a1-45bc-a8c0-248a57b78e10-maas-api-service-ca" pod "156f1da2-39a1-45bc-a8c0-248a57b78e10" (UID: "156f1da2-39a1-45bc-a8c0-248a57b78e10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:45:18.642656 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.642643    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/156f1da2-39a1-45bc-a8c0-248a57b78e10-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:45:18.644200 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.644181    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156f1da2-39a1-45bc-a8c0-248a57b78e10-kube-api-access-64rsm" pod "156f1da2-39a1-45bc-a8c0-248a57b78e10" (UID: "156f1da2-39a1-45bc-a8c0-248a57b78e10"). InnerVolumeSpecName "kube-api-access-64rsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:45:18.743881 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.743850    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f1b525-8ff0-448c-aba4-333082f26bf9-kube-api-access-rt8kq\" (UniqueName: \"kubernetes.io/projected/28f1b525-8ff0-448c-aba4-333082f26bf9-kube-api-access-rt8kq\") pod \"28f1b525-8ff0-448c-aba4-333082f26bf9\" (UID: \"28f1b525-8ff0-448c-aba4-333082f26bf9\") "
Nov 02 23:45:18.744066 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.743890    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed09312-d986-44b0-8f95-92349baa7d36-kube-api-access-grvcc\" (UniqueName: \"kubernetes.io/projected/8ed09312-d986-44b0-8f95-92349baa7d36-kube-api-access-grvcc\") pod \"8ed09312-d986-44b0-8f95-92349baa7d36\" (UID: \"8ed09312-d986-44b0-8f95-92349baa7d36\") "
Nov 02 23:45:18.744066 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.743920    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ed09312-d986-44b0-8f95-92349baa7d36-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ed09312-d986-44b0-8f95-92349baa7d36-maas-api-service-ca\") pod \"8ed09312-d986-44b0-8f95-92349baa7d36\" (UID: \"8ed09312-d986-44b0-8f95-92349baa7d36\") "
Nov 02 23:45:18.744066 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.743963    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28f1b525-8ff0-448c-aba4-333082f26bf9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f1b525-8ff0-448c-aba4-333082f26bf9-maas-api-service-ca\") pod \"28f1b525-8ff0-448c-aba4-333082f26bf9\" (UID: \"28f1b525-8ff0-448c-aba4-333082f26bf9\") "
Nov 02 23:45:18.744330 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.744299    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed09312-d986-44b0-8f95-92349baa7d36-maas-api-service-ca" pod "8ed09312-d986-44b0-8f95-92349baa7d36" (UID: "8ed09312-d986-44b0-8f95-92349baa7d36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:45:18.744432 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.744415    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f1b525-8ff0-448c-aba4-333082f26bf9-maas-api-service-ca" pod "28f1b525-8ff0-448c-aba4-333082f26bf9" (UID: "28f1b525-8ff0-448c-aba4-333082f26bf9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:45:18.744587 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.744566    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64rsm\" (UniqueName: \"kubernetes.io/projected/156f1da2-39a1-45bc-a8c0-248a57b78e10-kube-api-access-64rsm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:45:18.744659 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.744592    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ed09312-d986-44b0-8f95-92349baa7d36-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:45:18.744659 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.744608    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f1b525-8ff0-448c-aba4-333082f26bf9-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:45:18.746477 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.746452    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed09312-d986-44b0-8f95-92349baa7d36-kube-api-access-grvcc" pod "8ed09312-d986-44b0-8f95-92349baa7d36" (UID: "8ed09312-d986-44b0-8f95-92349baa7d36"). InnerVolumeSpecName "kube-api-access-grvcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:45:18.746587 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.746487    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f1b525-8ff0-448c-aba4-333082f26bf9-kube-api-access-rt8kq" pod "28f1b525-8ff0-448c-aba4-333082f26bf9" (UID: "28f1b525-8ff0-448c-aba4-333082f26bf9"). InnerVolumeSpecName "kube-api-access-rt8kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:45:18.845687 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.845655    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt8kq\" (UniqueName: \"kubernetes.io/projected/28f1b525-8ff0-448c-aba4-333082f26bf9-kube-api-access-rt8kq\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:45:18.845687 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:18.845681    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grvcc\" (UniqueName: \"kubernetes.io/projected/8ed09312-d986-44b0-8f95-92349baa7d36-kube-api-access-grvcc\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:45:19.362942 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.362916    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc"
Nov 02 23:45:19.363341 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.362918    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc" event={"ID":"28f1b525-8ff0-448c-aba4-333082f26bf9","Type":"ContainerDied","Data":"0a63d1863544ba354b769e1818ae25000b9d5096ef1a65089ec10c4453ac3590"}
Nov 02 23:45:19.363341 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.363120    4294 scope.go:122] "RemoveContainer" containerID="7d256df3f110779ff2de5571a6ac89272b2260557c01f97ba3a070335276bc77"
Nov 02 23:45:19.364500 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.364478    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55"
Nov 02 23:45:19.364604 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.364490    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55" event={"ID":"156f1da2-39a1-45bc-a8c0-248a57b78e10","Type":"ContainerDied","Data":"bd4eb146eccc3aaa57a0db2ba4f1e9df21de17e2513bb6a7d3767ab6b06810f6"}
Nov 02 23:45:19.366040 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.366020    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf" event={"ID":"8ed09312-d986-44b0-8f95-92349baa7d36","Type":"ContainerDied","Data":"b83b55329232a709732c388a73ddd7f798364d8482e4114e6c55cb3262725d84"}
Nov 02 23:45:19.366137 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.366076    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf"
Nov 02 23:45:19.376881 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.376799    4294 scope.go:122] "RemoveContainer" containerID="6db813560369100841cce694fff6269dd45973e8e696ce0a09b390ae7f3c3d49"
Nov 02 23:45:19.391137 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.391118    4294 scope.go:122] "RemoveContainer" containerID="1d3db72258cf290976723e9b07c95fe92e1dc20fa35adb450ef5937ae2fc636f"
Nov 02 23:45:19.520348 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.520325    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc"
Nov 02 23:45:19.586097 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.586061    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf"]
Nov 02 23:45:19.587666 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.587644    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ba1918-29894385-cdxpf"]
Nov 02 23:45:19.656159 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.656093    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbe05a3f-2410-4cbc-adab-33b1fa3625e3-kube-api-access-r57tx\" (UniqueName: \"kubernetes.io/projected/bbe05a3f-2410-4cbc-adab-33b1fa3625e3-kube-api-access-r57tx\") pod \"bbe05a3f-2410-4cbc-adab-33b1fa3625e3\" (UID: \"bbe05a3f-2410-4cbc-adab-33b1fa3625e3\") "
Nov 02 23:45:19.656275 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.656193    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe05a3f-2410-4cbc-adab-33b1fa3625e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe05a3f-2410-4cbc-adab-33b1fa3625e3-maas-api-service-ca\") pod \"bbe05a3f-2410-4cbc-adab-33b1fa3625e3\" (UID: \"bbe05a3f-2410-4cbc-adab-33b1fa3625e3\") "
Nov 02 23:45:19.656520 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.656493    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe05a3f-2410-4cbc-adab-33b1fa3625e3-maas-api-service-ca" pod "bbe05a3f-2410-4cbc-adab-33b1fa3625e3" (UID: "bbe05a3f-2410-4cbc-adab-33b1fa3625e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:45:19.656683 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.656666    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe05a3f-2410-4cbc-adab-33b1fa3625e3-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:45:19.658217 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.658200    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe05a3f-2410-4cbc-adab-33b1fa3625e3-kube-api-access-r57tx" pod "bbe05a3f-2410-4cbc-adab-33b1fa3625e3" (UID: "bbe05a3f-2410-4cbc-adab-33b1fa3625e3"). InnerVolumeSpecName "kube-api-access-r57tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:45:19.687613 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.687588    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc"]
Nov 02 23:45:19.697008 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.696970    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-28f82c-29894385-h5rbc"]
Nov 02 23:45:19.757833 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.757807    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r57tx\" (UniqueName: \"kubernetes.io/projected/bbe05a3f-2410-4cbc-adab-33b1fa3625e3-kube-api-access-r57tx\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:45:19.804295 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.804270    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55"]
Nov 02 23:45:19.814387 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:19.814362    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894385-5xh55"]
Nov 02 23:45:20.375623 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:20.375592    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc"
Nov 02 23:45:20.375623 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:20.375615    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc" event={"ID":"bbe05a3f-2410-4cbc-adab-33b1fa3625e3","Type":"ContainerDied","Data":"339db66f830efc3876f9518477a47e5d38a54418fffc81c76995e213efac5534"}
Nov 02 23:45:20.376138 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:20.375674    4294 scope.go:122] "RemoveContainer" containerID="816c3b349bbf95ae45d51fb1043451b62f45427e260dac0195ec3fa32770cc46"
Nov 02 23:45:20.534449 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:20.534419    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc"]
Nov 02 23:45:20.537338 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:20.537310    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3d8404-29894385-d2fwc"]
Nov 02 23:45:20.548424 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:20.548399    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="156f1da2-39a1-45bc-a8c0-248a57b78e10" path="/var/lib/kubelet/pods/156f1da2-39a1-45bc-a8c0-248a57b78e10/volumes"
Nov 02 23:45:20.548811 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:20.548798    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f1b525-8ff0-448c-aba4-333082f26bf9" path="/var/lib/kubelet/pods/28f1b525-8ff0-448c-aba4-333082f26bf9/volumes"
Nov 02 23:45:20.549198 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:20.549185    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ed09312-d986-44b0-8f95-92349baa7d36" path="/var/lib/kubelet/pods/8ed09312-d986-44b0-8f95-92349baa7d36/volumes"
Nov 02 23:45:20.549843 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:20.549829    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbe05a3f-2410-4cbc-adab-33b1fa3625e3" path="/var/lib/kubelet/pods/bbe05a3f-2410-4cbc-adab-33b1fa3625e3/volumes"
Nov 02 23:45:21.052222 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:21.052193    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:45:25.552605 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:25.552575    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:45:30.030162 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.030133    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:45:30.967544 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.967510    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"]
Nov 02 23:45:30.968112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968098    4294 cpu_manager.go:414] "containerMap: removing container" podUID="156f1da2-39a1-45bc-a8c0-248a57b78e10" containerName="cleanup"
Nov 02 23:45:30.968171 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968113    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156f1da2-39a1-45bc-a8c0-248a57b78e10" containerName="cleanup"
Nov 02 23:45:30.968171 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968125    4294 cpu_manager.go:414] "containerMap: removing container" podUID="bbe05a3f-2410-4cbc-adab-33b1fa3625e3" containerName="cleanup"
Nov 02 23:45:30.968171 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968130    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe05a3f-2410-4cbc-adab-33b1fa3625e3" containerName="cleanup"
Nov 02 23:45:30.968171 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968137    4294 cpu_manager.go:414] "containerMap: removing container" podUID="28f1b525-8ff0-448c-aba4-333082f26bf9" containerName="cleanup"
Nov 02 23:45:30.968171 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968142    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f1b525-8ff0-448c-aba4-333082f26bf9" containerName="cleanup"
Nov 02 23:45:30.968171 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968150    4294 cpu_manager.go:414] "containerMap: removing container" podUID="28f1b525-8ff0-448c-aba4-333082f26bf9" containerName="cleanup"
Nov 02 23:45:30.968171 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968154    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f1b525-8ff0-448c-aba4-333082f26bf9" containerName="cleanup"
Nov 02 23:45:30.968171 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968173    4294 cpu_manager.go:414] "containerMap: removing container" podUID="bbe05a3f-2410-4cbc-adab-33b1fa3625e3" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968178    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe05a3f-2410-4cbc-adab-33b1fa3625e3" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968184    4294 cpu_manager.go:414] "containerMap: removing container" podUID="8ed09312-d986-44b0-8f95-92349baa7d36" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968189    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed09312-d986-44b0-8f95-92349baa7d36" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968198    4294 cpu_manager.go:414] "containerMap: removing container" podUID="28f1b525-8ff0-448c-aba4-333082f26bf9" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968202    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f1b525-8ff0-448c-aba4-333082f26bf9" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968217    4294 cpu_manager.go:414] "containerMap: removing container" podUID="156f1da2-39a1-45bc-a8c0-248a57b78e10" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968221    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156f1da2-39a1-45bc-a8c0-248a57b78e10" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968234    4294 cpu_manager.go:414] "containerMap: removing container" podUID="8ed09312-d986-44b0-8f95-92349baa7d36" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968238    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed09312-d986-44b0-8f95-92349baa7d36" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968244    4294 cpu_manager.go:414] "containerMap: removing container" podUID="8ed09312-d986-44b0-8f95-92349baa7d36" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968248    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed09312-d986-44b0-8f95-92349baa7d36" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968340    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe05a3f-2410-4cbc-adab-33b1fa3625e3" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968347    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed09312-d986-44b0-8f95-92349baa7d36" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968355    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f1b525-8ff0-448c-aba4-333082f26bf9" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968361    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe05a3f-2410-4cbc-adab-33b1fa3625e3" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968369    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="156f1da2-39a1-45bc-a8c0-248a57b78e10" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968374    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed09312-d986-44b0-8f95-92349baa7d36" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968380    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed09312-d986-44b0-8f95-92349baa7d36" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968385    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="156f1da2-39a1-45bc-a8c0-248a57b78e10" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968390    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f1b525-8ff0-448c-aba4-333082f26bf9" containerName="cleanup"
Nov 02 23:45:30.968431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.968395    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f1b525-8ff0-448c-aba4-333082f26bf9" containerName="cleanup"
Nov 02 23:45:30.972771 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:30.972756    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:31.023667 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.023636    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"]
Nov 02 23:45:31.067528 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.067503    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-a51378\"/\"kube-root-ca.crt\""
Nov 02 23:45:31.072586 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.072569    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-a51378\"/\"auth-test-model-a51378-kserve-self-signed-certs\""
Nov 02 23:45:31.072665 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.072570    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-a51378\"/\"openshift-service-ca.crt\""
Nov 02 23:45:31.072665 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.072570    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-a51378\"/\"default-dockercfg-jxwpt\""
Nov 02 23:45:31.077534 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.077515    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:45:31.085860 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.085843    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-home\") pod \"auth-test-model-a51378-kserve-84b565f7cc-4vw4f\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") " pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:31.085962 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.085948    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-tmp-dir\") pod \"auth-test-model-a51378-kserve-84b565f7cc-4vw4f\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") " pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:31.086028 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.086007    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-dshm\") pod \"auth-test-model-a51378-kserve-84b565f7cc-4vw4f\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") " pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:31.086072 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.086032    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-model-cache\") pod \"auth-test-model-a51378-kserve-84b565f7cc-4vw4f\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") " pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:31.086122 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.086109    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn9b2\" (UniqueName: \"kubernetes.io/projected/906b9443-bdbf-49cc-8abb-d67fd25850c7-kube-api-access-dn9b2\") pod \"auth-test-model-a51378-kserve-84b565f7cc-4vw4f\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") " pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:31.086169 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.086136    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/906b9443-bdbf-49cc-8abb-d67fd25850c7-tls-certs\") pod \"auth-test-model-a51378-kserve-84b565f7cc-4vw4f\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") " pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:31.187444 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.187411    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-home\") pod \"auth-test-model-a51378-kserve-84b565f7cc-4vw4f\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") " pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:31.187657 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.187473    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-tmp-dir\") pod \"auth-test-model-a51378-kserve-84b565f7cc-4vw4f\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") " pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:31.187657 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.187502    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-dshm\") pod \"auth-test-model-a51378-kserve-84b565f7cc-4vw4f\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") " pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:31.187657 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.187521    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-model-cache\") pod \"auth-test-model-a51378-kserve-84b565f7cc-4vw4f\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") " pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:31.187657 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.187546    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn9b2\" (UniqueName: \"kubernetes.io/projected/906b9443-bdbf-49cc-8abb-d67fd25850c7-kube-api-access-dn9b2\") pod \"auth-test-model-a51378-kserve-84b565f7cc-4vw4f\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") " pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:31.187657 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.187575    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/906b9443-bdbf-49cc-8abb-d67fd25850c7-tls-certs\") pod \"auth-test-model-a51378-kserve-84b565f7cc-4vw4f\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") " pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:31.188325 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.188295    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-home\") pod \"auth-test-model-a51378-kserve-84b565f7cc-4vw4f\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") " pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:31.188504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.188318    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-model-cache\") pod \"auth-test-model-a51378-kserve-84b565f7cc-4vw4f\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") " pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:31.188603 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.188357    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-tmp-dir\") pod \"auth-test-model-a51378-kserve-84b565f7cc-4vw4f\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") " pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:31.198810 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.198780    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-dshm\") pod \"auth-test-model-a51378-kserve-84b565f7cc-4vw4f\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") " pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:31.200148 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.200126    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/906b9443-bdbf-49cc-8abb-d67fd25850c7-tls-certs\") pod \"auth-test-model-a51378-kserve-84b565f7cc-4vw4f\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") " pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:31.242332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.242306    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn9b2\" (UniqueName: \"kubernetes.io/projected/906b9443-bdbf-49cc-8abb-d67fd25850c7-kube-api-access-dn9b2\") pod \"auth-test-model-a51378-kserve-84b565f7cc-4vw4f\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") " pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:31.285023 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.284997    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:31.652468 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:31.652445    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"]
Nov 02 23:45:31.653387 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:45:31.653356    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod906b9443_bdbf_49cc_8abb_d67fd25850c7.slice/crio-1e6d3ffdbcca7a294139eec274611979dd9cc706a94a118ab034aafa43e2e1ef WatchSource:0}: Error finding container 1e6d3ffdbcca7a294139eec274611979dd9cc706a94a118ab034aafa43e2e1ef: Status 404 returned error can't find the container with id 1e6d3ffdbcca7a294139eec274611979dd9cc706a94a118ab034aafa43e2e1ef
Nov 02 23:45:32.454022 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:32.453963    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f" event={"ID":"906b9443-bdbf-49cc-8abb-d67fd25850c7","Type":"ContainerStarted","Data":"66d0ecdb79065b161bc2516d65a775b05d73ff958e21a5e268b526d147aa89e6"}
Nov 02 23:45:32.454411 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:32.454032    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f" event={"ID":"906b9443-bdbf-49cc-8abb-d67fd25850c7","Type":"ContainerStarted","Data":"1e6d3ffdbcca7a294139eec274611979dd9cc706a94a118ab034aafa43e2e1ef"}
Nov 02 23:45:32.564810 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:32.564766    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f" podStartSLOduration=2.564752543 podStartE2EDuration="2.564752543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:32.562408268 +0000 UTC m=+3172.130763055" watchObservedRunningTime="2026-11-02 23:45:32.564752543 +0000 UTC m=+3172.133107330"
Nov 02 23:45:33.285916 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:33.285880    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:34.651884 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:34.651851    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:45:39.104811 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:39.104776    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:45:39.503130 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:39.503096    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:45:39.672893 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:39.672858    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:45:40.463608 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.463571    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"]
Nov 02 23:45:40.464415 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.464392    4294 cpu_manager.go:414] "containerMap: removing container" podUID="bbe05a3f-2410-4cbc-adab-33b1fa3625e3" containerName="cleanup"
Nov 02 23:45:40.464544 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.464415    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe05a3f-2410-4cbc-adab-33b1fa3625e3" containerName="cleanup"
Nov 02 23:45:40.464544 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.464477    4294 cpu_manager.go:414] "containerMap: removing container" podUID="156f1da2-39a1-45bc-a8c0-248a57b78e10" containerName="cleanup"
Nov 02 23:45:40.464544 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.464486    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156f1da2-39a1-45bc-a8c0-248a57b78e10" containerName="cleanup"
Nov 02 23:45:40.464740 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.464641    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="156f1da2-39a1-45bc-a8c0-248a57b78e10" containerName="cleanup"
Nov 02 23:45:40.464740 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.464666    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe05a3f-2410-4cbc-adab-33b1fa3625e3" containerName="cleanup"
Nov 02 23:45:40.468280 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.468260    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:40.522242 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.522210    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"]
Nov 02 23:45:40.563809 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.563779    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7ca60f\"/\"kube-root-ca.crt\""
Nov 02 23:45:40.563809 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.563780    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7ca60f\"/\"openshift-service-ca.crt\""
Nov 02 23:45:40.564019 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.563829    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7ca60f\"/\"auth-test-model-7ca60f-kserve-self-signed-certs\""
Nov 02 23:45:40.566139 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.566118    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7ca60f\"/\"default-dockercfg-j88dc\""
Nov 02 23:45:40.586772 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.586749    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:45:40.598149 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.598127    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-tls-certs\") pod \"auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:40.598270 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.598155    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw58q\" (UniqueName: \"kubernetes.io/projected/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-kube-api-access-pw58q\") pod \"auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:40.598270 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.598184    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-dshm\") pod \"auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:40.598357 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.598327    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-model-cache\") pod \"auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:40.598466 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.598449    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-home\") pod \"auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:40.598552 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.598538    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-tmp-dir\") pod \"auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:40.699341 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.699304    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-home\") pod \"auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:40.699529 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.699397    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-tmp-dir\") pod \"auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:40.699529 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.699458    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-tls-certs\") pod \"auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:40.699529 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.699475    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw58q\" (UniqueName: \"kubernetes.io/projected/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-kube-api-access-pw58q\") pod \"auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:40.699529 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.699506    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-dshm\") pod \"auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:40.699778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.699550    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-model-cache\") pod \"auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:40.699778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.699703    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-home\") pod \"auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:40.699890 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.699794    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-tmp-dir\") pod \"auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:40.700035 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.700013    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-model-cache\") pod \"auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:40.701851 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.701819    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-dshm\") pod \"auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:40.702105 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.702087    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-tls-certs\") pod \"auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:40.753582 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.753561    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw58q\" (UniqueName: \"kubernetes.io/projected/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-kube-api-access-pw58q\") pod \"auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:40.781230 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:40.781210    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:41.148304 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:45:41.148271    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cef90d4_bdb1_4b9b_a925_742b2d29ee0e.slice/crio-0ee982c6fad08bbf44b54674bda4c3108616a463e85d30ecc779baffc643bc1e WatchSource:0}: Error finding container 0ee982c6fad08bbf44b54674bda4c3108616a463e85d30ecc779baffc643bc1e: Status 404 returned error can't find the container with id 0ee982c6fad08bbf44b54674bda4c3108616a463e85d30ecc779baffc643bc1e
Nov 02 23:45:41.148866 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:41.148845    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"]
Nov 02 23:45:41.285800 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:41.285774    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:41.299185 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:41.299161    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:41.518805 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:41.518773    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl" event={"ID":"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e","Type":"ContainerStarted","Data":"916a3d5ee19a8c87208dd9f625b86337e86dd887a57b0d2048793bd6e1ab44ab"}
Nov 02 23:45:41.519190 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:41.518814    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl" event={"ID":"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e","Type":"ContainerStarted","Data":"0ee982c6fad08bbf44b54674bda4c3108616a463e85d30ecc779baffc643bc1e"}
Nov 02 23:45:41.530213 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:41.530192    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:45:41.631504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:41.631449    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl" podStartSLOduration=1.631436242 podStartE2EDuration="1.631436242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:45:41.627941555 +0000 UTC m=+3181.196296336" watchObservedRunningTime="2026-11-02 23:45:41.631436242 +0000 UTC m=+3181.199791028"
Nov 02 23:45:41.768205 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:45:41.768168    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:45:41.781452 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:41.781418    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:43.178320 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:43.178287    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:45:44.343033 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:44.342969    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:45:48.943551 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:48.943515    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:45:50.782125 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:50.782091    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:50.795614 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:50.795589    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:51.591855 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:51.591827    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:45:53.632760 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:53.632726    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:45:53.812251 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:53.812175    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:45:58.075887 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:45:58.075855    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:46:02.664360 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:46:02.664328    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:46:06.246682 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:46:06.246641    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:46:07.153763 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:46:07.153729    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:46:11.739732 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:46:11.739700    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:46:11.753763 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:46:11.753728    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:46:16.352618 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:46:16.352588    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:46:17.298423 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:46:17.298378    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:46:20.935615 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:46:20.935582    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:46:25.472862 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:46:25.472823    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:46:30.156288 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:46:30.156247    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:46:32.041154 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:46:32.041120    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:46:34.815362 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:46:34.815310    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:46:37.076186 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:46:37.076153    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:46:39.351969 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:46:39.351935    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:46:39.931594 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:46:39.931558    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:46:41.749700 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:46:41.749657    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:46:43.845492 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:46:43.845455    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:46:48.343464 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:46:48.343423    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:46:52.953082 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:46:52.953017    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:46:57.468555 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:46:57.468514    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:01.840555 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:01.840498    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:02.009773 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:02.009694    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:06.474341 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:06.474306    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:11.040312 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:11.040272    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:11.754782 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:47:11.754739    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:47:14.001197 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:14.001163    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:14.531974 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:14.531946    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4"]
Nov 02 23:47:14.807042 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:14.806936    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:14.972377 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:14.972343    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:15.832633 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:15.832602    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:19.532745 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:19.532687    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4" podUID="28664c33-54e8-4ced-9a16-dc46980a8bf5" containerName="payload-pre-processing" containerID="cri-o://2f4cb97518db03d2a873358259d1722c06575dfdc7dbcff413c8805bd1add3e9" gracePeriod=25
Nov 02 23:47:19.547323 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:19.547294    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:19.804013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:19.803960    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4"
Nov 02 23:47:19.904674 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:19.904640    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28664c33-54e8-4ced-9a16-dc46980a8bf5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28664c33-54e8-4ced-9a16-dc46980a8bf5-plugins-config-volume\") pod \"28664c33-54e8-4ced-9a16-dc46980a8bf5\" (UID: \"28664c33-54e8-4ced-9a16-dc46980a8bf5\") "
Nov 02 23:47:19.904844 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:19.904730    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28664c33-54e8-4ced-9a16-dc46980a8bf5-kube-api-access-ch7r5\" (UniqueName: \"kubernetes.io/projected/28664c33-54e8-4ced-9a16-dc46980a8bf5-kube-api-access-ch7r5\") pod \"28664c33-54e8-4ced-9a16-dc46980a8bf5\" (UID: \"28664c33-54e8-4ced-9a16-dc46980a8bf5\") "
Nov 02 23:47:19.905017 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:19.904996    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28664c33-54e8-4ced-9a16-dc46980a8bf5-plugins-config-volume" pod "28664c33-54e8-4ced-9a16-dc46980a8bf5" (UID: "28664c33-54e8-4ced-9a16-dc46980a8bf5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:47:19.905218 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:19.905204    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28664c33-54e8-4ced-9a16-dc46980a8bf5-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:47:19.906772 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:19.906750    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28664c33-54e8-4ced-9a16-dc46980a8bf5-kube-api-access-ch7r5" pod "28664c33-54e8-4ced-9a16-dc46980a8bf5" (UID: "28664c33-54e8-4ced-9a16-dc46980a8bf5"). InnerVolumeSpecName "kube-api-access-ch7r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:47:20.006818 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:20.006778    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch7r5\" (UniqueName: \"kubernetes.io/projected/28664c33-54e8-4ced-9a16-dc46980a8bf5-kube-api-access-ch7r5\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:47:20.134771 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:20.134685    4294 generic.go:356] "Generic (PLEG): container finished" podID="28664c33-54e8-4ced-9a16-dc46980a8bf5" containerID="2f4cb97518db03d2a873358259d1722c06575dfdc7dbcff413c8805bd1add3e9" exitCode=0
Nov 02 23:47:20.134771 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:20.134735    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4"
Nov 02 23:47:20.134953 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:20.134779    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4" event={"ID":"28664c33-54e8-4ced-9a16-dc46980a8bf5","Type":"ContainerDied","Data":"2f4cb97518db03d2a873358259d1722c06575dfdc7dbcff413c8805bd1add3e9"}
Nov 02 23:47:20.134953 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:20.134839    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4" event={"ID":"28664c33-54e8-4ced-9a16-dc46980a8bf5","Type":"ContainerDied","Data":"87f515c5d59ad95e0bc6fe1689756f345662d5fe67277a8c87dbeeced93ca021"}
Nov 02 23:47:20.134953 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:20.134864    4294 scope.go:122] "RemoveContainer" containerID="2f4cb97518db03d2a873358259d1722c06575dfdc7dbcff413c8805bd1add3e9"
Nov 02 23:47:20.144958 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:20.144939    4294 scope.go:122] "RemoveContainer" containerID="2f4cb97518db03d2a873358259d1722c06575dfdc7dbcff413c8805bd1add3e9"
Nov 02 23:47:20.145218 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:47:20.145199    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f4cb97518db03d2a873358259d1722c06575dfdc7dbcff413c8805bd1add3e9\": container with ID starting with 2f4cb97518db03d2a873358259d1722c06575dfdc7dbcff413c8805bd1add3e9 not found: ID does not exist" containerID="2f4cb97518db03d2a873358259d1722c06575dfdc7dbcff413c8805bd1add3e9"
Nov 02 23:47:20.145268 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:20.145225    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4cb97518db03d2a873358259d1722c06575dfdc7dbcff413c8805bd1add3e9"} err="failed to get container status \"2f4cb97518db03d2a873358259d1722c06575dfdc7dbcff413c8805bd1add3e9\": rpc error: code = NotFound desc = could not find container \"2f4cb97518db03d2a873358259d1722c06575dfdc7dbcff413c8805bd1add3e9\": container with ID starting with 2f4cb97518db03d2a873358259d1722c06575dfdc7dbcff413c8805bd1add3e9 not found: ID does not exist"
Nov 02 23:47:20.292126 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:20.292096    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4"]
Nov 02 23:47:20.295837 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:20.295815    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-52dd59-54d44cb8c9-xf5v4"]
Nov 02 23:47:20.454177 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:20.454091    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:20.548237 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:20.548208    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28664c33-54e8-4ced-9a16-dc46980a8bf5" path="/var/lib/kubelet/pods/28664c33-54e8-4ced-9a16-dc46980a8bf5/volumes"
Nov 02 23:47:24.951496 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:24.951418    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:29.451857 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:29.451823    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:33.955493 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:33.955458    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:38.615670 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:38.615638    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:40.171724 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:40.171688    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:41.751465 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:47:41.751418    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:47:43.156411 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:43.156367    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:47.083531 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:47.083497    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:48.766871 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:48.766829    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:50.910283 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:50.910246    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:51.075940 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:51.075906    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:52.432512 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:52.432480    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:53.046663 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:53.046630    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:53.465801 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:53.465724    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:53.656598 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:53.656571    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:54.197421 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:54.197385    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:54.584600 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:54.584567    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:55.770226 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:55.770194    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:55.945696 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:55.945662    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:56.123186 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:56.123109    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:47:58.243681 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:47:58.243647    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:02.843015 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:02.842965    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:07.671857 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:07.671826    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:09.638809 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:09.638775    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:09.935313 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:09.935235    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:11.751944 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:48:11.751907    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:48:12.170441 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:12.170362    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:15.356947 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:15.356904    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:16.664993 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:16.664954    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:21.258304 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:21.258271    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:25.788277 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:25.788237    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:29.368182 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:29.368150    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:29.526791 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:29.526760    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:30.260836 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:30.260801    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:34.751536 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:34.751504    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:39.247113 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:39.247039    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:40.486334 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:40.486295    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:41.333934 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:41.333892    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"]
Nov 02 23:48:41.334226 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:41.334129    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr" podUID="4571d9d7-7554-444c-a4d7-ba0daf1006fc" containerName="istio-proxy" containerID="cri-o://6efdc662aac95b84e6ff6b739b6eba3e81dcc129866440e5b65357be01202897" gracePeriod=30
Nov 02 23:48:41.337273 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:41.337248    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"]
Nov 02 23:48:41.337446 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:41.337428    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr" podUID="d078f228-be56-497c-87f6-8e94b080770c" containerName="istio-proxy" containerID="cri-o://66429b844d4597187d38d95933bfc0eacd10d269a218a526444506c0468730d1" gracePeriod=30
Nov 02 23:48:41.473663 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:41.473627    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"]
Nov 02 23:48:41.473834 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:41.473818    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5" podUID="fdbd38f7-a556-4fdf-acfc-9077aac74ec7" containerName="istio-proxy" containerID="cri-o://602bd9a14dc375db51bcdca7e2894637533b92e7ea7c6f2da44b005cf4b091cc" gracePeriod=30
Nov 02 23:48:41.750390 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:48:41.750353    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:48:43.892157 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:43.892120    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:43.922341 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:43.922310    4294 patch_prober.go:29] interesting pod/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 23:48:43.922467 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:43.922359    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr" podUID="d078f228-be56-497c-87f6-8e94b080770c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 23:48:44.235428 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:44.235348    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"]
Nov 02 23:48:44.283806 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:44.283775    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:44.806179 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:44.806144    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 23:48:44.806356 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:44.806200    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5" podUID="fdbd38f7-a556-4fdf-acfc-9077aac74ec7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 23:48:46.623053 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.623023    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:48:46.630283 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.630263    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:48:46.668272 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.668242    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-token\" (UniqueName: \"kubernetes.io/projected/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-token\") pod \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") "
Nov 02 23:48:46.668415 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.668314    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d078f228-be56-497c-87f6-8e94b080770c-kube-api-access-d7mpp\" (UniqueName: \"kubernetes.io/projected/d078f228-be56-497c-87f6-8e94b080770c-kube-api-access-d7mpp\") pod \"d078f228-be56-497c-87f6-8e94b080770c\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") "
Nov 02 23:48:46.668415 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.668365    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-envoy\") pod \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") "
Nov 02 23:48:46.668544 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.668421    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-podinfo\") pod \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") "
Nov 02 23:48:46.668544 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.668465    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d078f228-be56-497c-87f6-8e94b080770c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d078f228-be56-497c-87f6-8e94b080770c-istio-podinfo\") pod \"d078f228-be56-497c-87f6-8e94b080770c\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") "
Nov 02 23:48:46.668544 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.668521    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-workload-certs\") pod \"d078f228-be56-497c-87f6-8e94b080770c\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") "
Nov 02 23:48:46.668715 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.668632    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-workload-socket\") pod \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") "
Nov 02 23:48:46.668715 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.668666    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-data\") pod \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") "
Nov 02 23:48:46.668715 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.668700    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-istio-envoy\") pod \"d078f228-be56-497c-87f6-8e94b080770c\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") "
Nov 02 23:48:46.668882 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.668746    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4571d9d7-7554-444c-a4d7-ba0daf1006fc-kube-api-access-m8v54\" (UniqueName: \"kubernetes.io/projected/4571d9d7-7554-444c-a4d7-ba0daf1006fc-kube-api-access-m8v54\") pod \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") "
Nov 02 23:48:46.668882 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.668794    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-istio-data\") pod \"d078f228-be56-497c-87f6-8e94b080770c\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") "
Nov 02 23:48:46.668967 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.668924    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-workload-certs\") pod \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") "
Nov 02 23:48:46.669044 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.668962    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istiod-ca-cert\") pod \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") "
Nov 02 23:48:46.669044 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.669007    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-workload-socket\") pod \"d078f228-be56-497c-87f6-8e94b080770c\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") "
Nov 02 23:48:46.669155 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.669056    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-credential-socket\") pod \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\" (UID: \"4571d9d7-7554-444c-a4d7-ba0daf1006fc\") "
Nov 02 23:48:46.669155 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.669109    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d078f228-be56-497c-87f6-8e94b080770c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d078f228-be56-497c-87f6-8e94b080770c-istiod-ca-cert\") pod \"d078f228-be56-497c-87f6-8e94b080770c\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") "
Nov 02 23:48:46.669155 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.669148    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-credential-socket\") pod \"d078f228-be56-497c-87f6-8e94b080770c\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") "
Nov 02 23:48:46.669326 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.669178    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d078f228-be56-497c-87f6-8e94b080770c-istio-token\" (UniqueName: \"kubernetes.io/projected/d078f228-be56-497c-87f6-8e94b080770c-istio-token\") pod \"d078f228-be56-497c-87f6-8e94b080770c\" (UID: \"d078f228-be56-497c-87f6-8e94b080770c\") "
Nov 02 23:48:46.669784 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.669729    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-workload-certs" pod "d078f228-be56-497c-87f6-8e94b080770c" (UID: "d078f228-be56-497c-87f6-8e94b080770c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:48:46.669784 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.669748    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-workload-socket" pod "d078f228-be56-497c-87f6-8e94b080770c" (UID: "d078f228-be56-497c-87f6-8e94b080770c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:48:46.670143 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.669962    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-istio-data" pod "d078f228-be56-497c-87f6-8e94b080770c" (UID: "d078f228-be56-497c-87f6-8e94b080770c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:48:46.670143 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.670032    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d078f228-be56-497c-87f6-8e94b080770c-istiod-ca-cert" pod "d078f228-be56-497c-87f6-8e94b080770c" (UID: "d078f228-be56-497c-87f6-8e94b080770c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:48:46.670281 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.670188    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-credential-socket" pod "4571d9d7-7554-444c-a4d7-ba0daf1006fc" (UID: "4571d9d7-7554-444c-a4d7-ba0daf1006fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:48:46.670281 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.670224    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-workload-certs" pod "4571d9d7-7554-444c-a4d7-ba0daf1006fc" (UID: "4571d9d7-7554-444c-a4d7-ba0daf1006fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:48:46.670561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.670411    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-data" pod "4571d9d7-7554-444c-a4d7-ba0daf1006fc" (UID: "4571d9d7-7554-444c-a4d7-ba0daf1006fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:48:46.670561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.670421    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istiod-ca-cert" pod "4571d9d7-7554-444c-a4d7-ba0daf1006fc" (UID: "4571d9d7-7554-444c-a4d7-ba0daf1006fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:48:46.670561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.670440    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-workload-socket" pod "4571d9d7-7554-444c-a4d7-ba0daf1006fc" (UID: "4571d9d7-7554-444c-a4d7-ba0daf1006fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:48:46.670907 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.670881    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-credential-socket" pod "d078f228-be56-497c-87f6-8e94b080770c" (UID: "d078f228-be56-497c-87f6-8e94b080770c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:48:46.672729 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.672700    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-envoy" pod "4571d9d7-7554-444c-a4d7-ba0daf1006fc" (UID: "4571d9d7-7554-444c-a4d7-ba0daf1006fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:48:46.672829 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.672765    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d078f228-be56-497c-87f6-8e94b080770c-kube-api-access-d7mpp" pod "d078f228-be56-497c-87f6-8e94b080770c" (UID: "d078f228-be56-497c-87f6-8e94b080770c"). InnerVolumeSpecName "kube-api-access-d7mpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:48:46.672829 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.672791    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4571d9d7-7554-444c-a4d7-ba0daf1006fc-kube-api-access-m8v54" pod "4571d9d7-7554-444c-a4d7-ba0daf1006fc" (UID: "4571d9d7-7554-444c-a4d7-ba0daf1006fc"). InnerVolumeSpecName "kube-api-access-m8v54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:48:46.673062 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.673041    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-token" pod "4571d9d7-7554-444c-a4d7-ba0daf1006fc" (UID: "4571d9d7-7554-444c-a4d7-ba0daf1006fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:48:46.673248 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.673222    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d078f228-be56-497c-87f6-8e94b080770c-istio-podinfo" pod "d078f228-be56-497c-87f6-8e94b080770c" (UID: "d078f228-be56-497c-87f6-8e94b080770c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:48:46.673848 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.673826    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-podinfo" pod "4571d9d7-7554-444c-a4d7-ba0daf1006fc" (UID: "4571d9d7-7554-444c-a4d7-ba0daf1006fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:48:46.674506 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.674483    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d078f228-be56-497c-87f6-8e94b080770c-istio-token" pod "d078f228-be56-497c-87f6-8e94b080770c" (UID: "d078f228-be56-497c-87f6-8e94b080770c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:48:46.674739 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.674710    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-istio-envoy" pod "d078f228-be56-497c-87f6-8e94b080770c" (UID: "d078f228-be56-497c-87f6-8e94b080770c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:48:46.687327 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.687137    4294 generic.go:356] "Generic (PLEG): container finished" podID="4571d9d7-7554-444c-a4d7-ba0daf1006fc" containerID="6efdc662aac95b84e6ff6b739b6eba3e81dcc129866440e5b65357be01202897" exitCode=0
Nov 02 23:48:46.687806 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.687739    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"
Nov 02 23:48:46.689189 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.689099    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr" event={"ID":"4571d9d7-7554-444c-a4d7-ba0daf1006fc","Type":"ContainerDied","Data":"6efdc662aac95b84e6ff6b739b6eba3e81dcc129866440e5b65357be01202897"}
Nov 02 23:48:46.689189 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.689153    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr" event={"ID":"4571d9d7-7554-444c-a4d7-ba0daf1006fc","Type":"ContainerDied","Data":"724d1881f6326cc1f707ac513ce245858cfafd6d74b09a0adab0ae470814c0fa"}
Nov 02 23:48:46.689189 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.689186    4294 scope.go:122] "RemoveContainer" containerID="6efdc662aac95b84e6ff6b739b6eba3e81dcc129866440e5b65357be01202897"
Nov 02 23:48:46.698247 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.698220    4294 generic.go:356] "Generic (PLEG): container finished" podID="d078f228-be56-497c-87f6-8e94b080770c" containerID="66429b844d4597187d38d95933bfc0eacd10d269a218a526444506c0468730d1" exitCode=0
Nov 02 23:48:46.698355 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.698267    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"
Nov 02 23:48:46.698355 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.698307    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr" event={"ID":"d078f228-be56-497c-87f6-8e94b080770c","Type":"ContainerDied","Data":"66429b844d4597187d38d95933bfc0eacd10d269a218a526444506c0468730d1"}
Nov 02 23:48:46.698485 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.698359    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr" event={"ID":"d078f228-be56-497c-87f6-8e94b080770c","Type":"ContainerDied","Data":"9b45939f7629f387d30d1fb3c02b43f5ff9cee9a736740138a23bcdb6c9cd189"}
Nov 02 23:48:46.701186 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.700406    4294 generic.go:356] "Generic (PLEG): container finished" podID="fdbd38f7-a556-4fdf-acfc-9077aac74ec7" containerID="602bd9a14dc375db51bcdca7e2894637533b92e7ea7c6f2da44b005cf4b091cc" exitCode=0
Nov 02 23:48:46.701186 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.700480    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5" event={"ID":"fdbd38f7-a556-4fdf-acfc-9077aac74ec7","Type":"ContainerDied","Data":"602bd9a14dc375db51bcdca7e2894637533b92e7ea7c6f2da44b005cf4b091cc"}
Nov 02 23:48:46.707776 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.707754    4294 scope.go:122] "RemoveContainer" containerID="e1ac82ff6b0c8f8219d1c78859909cddb14aefb2fd130246557b889e7dd143a5"
Nov 02 23:48:46.716871 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.716852    4294 scope.go:122] "RemoveContainer" containerID="6efdc662aac95b84e6ff6b739b6eba3e81dcc129866440e5b65357be01202897"
Nov 02 23:48:46.717170 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:48:46.717151    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6efdc662aac95b84e6ff6b739b6eba3e81dcc129866440e5b65357be01202897\": container with ID starting with 6efdc662aac95b84e6ff6b739b6eba3e81dcc129866440e5b65357be01202897 not found: ID does not exist" containerID="6efdc662aac95b84e6ff6b739b6eba3e81dcc129866440e5b65357be01202897"
Nov 02 23:48:46.717250 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.717177    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6efdc662aac95b84e6ff6b739b6eba3e81dcc129866440e5b65357be01202897"} err="failed to get container status \"6efdc662aac95b84e6ff6b739b6eba3e81dcc129866440e5b65357be01202897\": rpc error: code = NotFound desc = could not find container \"6efdc662aac95b84e6ff6b739b6eba3e81dcc129866440e5b65357be01202897\": container with ID starting with 6efdc662aac95b84e6ff6b739b6eba3e81dcc129866440e5b65357be01202897 not found: ID does not exist"
Nov 02 23:48:46.717250 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.717194    4294 scope.go:122] "RemoveContainer" containerID="e1ac82ff6b0c8f8219d1c78859909cddb14aefb2fd130246557b889e7dd143a5"
Nov 02 23:48:46.717419 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:48:46.717401    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ac82ff6b0c8f8219d1c78859909cddb14aefb2fd130246557b889e7dd143a5\": container with ID starting with e1ac82ff6b0c8f8219d1c78859909cddb14aefb2fd130246557b889e7dd143a5 not found: ID does not exist" containerID="e1ac82ff6b0c8f8219d1c78859909cddb14aefb2fd130246557b889e7dd143a5"
Nov 02 23:48:46.717479 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.717423    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ac82ff6b0c8f8219d1c78859909cddb14aefb2fd130246557b889e7dd143a5"} err="failed to get container status \"e1ac82ff6b0c8f8219d1c78859909cddb14aefb2fd130246557b889e7dd143a5\": rpc error: code = NotFound desc = could not find container \"e1ac82ff6b0c8f8219d1c78859909cddb14aefb2fd130246557b889e7dd143a5\": container with ID starting with e1ac82ff6b0c8f8219d1c78859909cddb14aefb2fd130246557b889e7dd143a5 not found: ID does not exist"
Nov 02 23:48:46.717479 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.717436    4294 scope.go:122] "RemoveContainer" containerID="66429b844d4597187d38d95933bfc0eacd10d269a218a526444506c0468730d1"
Nov 02 23:48:46.727012 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.726994    4294 scope.go:122] "RemoveContainer" containerID="66429b844d4597187d38d95933bfc0eacd10d269a218a526444506c0468730d1"
Nov 02 23:48:46.727319 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:48:46.727292    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66429b844d4597187d38d95933bfc0eacd10d269a218a526444506c0468730d1\": container with ID starting with 66429b844d4597187d38d95933bfc0eacd10d269a218a526444506c0468730d1 not found: ID does not exist" containerID="66429b844d4597187d38d95933bfc0eacd10d269a218a526444506c0468730d1"
Nov 02 23:48:46.727400 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.727333    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66429b844d4597187d38d95933bfc0eacd10d269a218a526444506c0468730d1"} err="failed to get container status \"66429b844d4597187d38d95933bfc0eacd10d269a218a526444506c0468730d1\": rpc error: code = NotFound desc = could not find container \"66429b844d4597187d38d95933bfc0eacd10d269a218a526444506c0468730d1\": container with ID starting with 66429b844d4597187d38d95933bfc0eacd10d269a218a526444506c0468730d1 not found: ID does not exist"
Nov 02 23:48:46.736466 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.736448    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:48:46.771549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.770438    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istiod-ca-cert\") pod \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") "
Nov 02 23:48:46.771549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.770470    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-podinfo\") pod \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") "
Nov 02 23:48:46.771549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.770614    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-workload-certs\") pod \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") "
Nov 02 23:48:46.771549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.770649    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-credential-socket\") pod \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") "
Nov 02 23:48:46.771549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.770761    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-envoy\") pod \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") "
Nov 02 23:48:46.771549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.770798    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-workload-socket\") pod \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") "
Nov 02 23:48:46.771549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.770837    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istiod-ca-cert" pod "fdbd38f7-a556-4fdf-acfc-9077aac74ec7" (UID: "fdbd38f7-a556-4fdf-acfc-9077aac74ec7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:48:46.771549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.770852    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-workload-certs" pod "fdbd38f7-a556-4fdf-acfc-9077aac74ec7" (UID: "fdbd38f7-a556-4fdf-acfc-9077aac74ec7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:48:46.771549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.770894    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-kube-api-access-7jgcz\" (UniqueName: \"kubernetes.io/projected/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-kube-api-access-7jgcz\") pod \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") "
Nov 02 23:48:46.771549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.770925    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-data\") pod \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") "
Nov 02 23:48:46.771549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.770960    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-credential-socket" pod "fdbd38f7-a556-4fdf-acfc-9077aac74ec7" (UID: "fdbd38f7-a556-4fdf-acfc-9077aac74ec7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:48:46.771549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.771007    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-token\" (UniqueName: \"kubernetes.io/projected/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-token\") pod \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\" (UID: \"fdbd38f7-a556-4fdf-acfc-9077aac74ec7\") "
Nov 02 23:48:46.771549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.771370    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-workload-socket" pod "fdbd38f7-a556-4fdf-acfc-9077aac74ec7" (UID: "fdbd38f7-a556-4fdf-acfc-9077aac74ec7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:48:46.771549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.771536    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-data" pod "fdbd38f7-a556-4fdf-acfc-9077aac74ec7" (UID: "fdbd38f7-a556-4fdf-acfc-9077aac74ec7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.771576    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.771596    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.771612    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d078f228-be56-497c-87f6-8e94b080770c-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.771626    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.771640    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.771654    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.772097    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.772119    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.772136    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.772151    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8v54\" (UniqueName: \"kubernetes.io/projected/4571d9d7-7554-444c-a4d7-ba0daf1006fc-kube-api-access-m8v54\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.772167    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.772181    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.772195    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.772210    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.772223    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.772237    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.772251    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4571d9d7-7554-444c-a4d7-ba0daf1006fc-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.772266    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d078f228-be56-497c-87f6-8e94b080770c-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.772280    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078f228-be56-497c-87f6-8e94b080770c-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.772295    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d078f228-be56-497c-87f6-8e94b080770c-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.772308    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4571d9d7-7554-444c-a4d7-ba0daf1006fc-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.772405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.772322    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7mpp\" (UniqueName: \"kubernetes.io/projected/d078f228-be56-497c-87f6-8e94b080770c-kube-api-access-d7mpp\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.773659 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.773072    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-envoy" pod "fdbd38f7-a556-4fdf-acfc-9077aac74ec7" (UID: "fdbd38f7-a556-4fdf-acfc-9077aac74ec7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:48:46.773659 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.773085    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-podinfo" pod "fdbd38f7-a556-4fdf-acfc-9077aac74ec7" (UID: "fdbd38f7-a556-4fdf-acfc-9077aac74ec7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:48:46.773659 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.773387    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-kube-api-access-7jgcz" pod "fdbd38f7-a556-4fdf-acfc-9077aac74ec7" (UID: "fdbd38f7-a556-4fdf-acfc-9077aac74ec7"). InnerVolumeSpecName "kube-api-access-7jgcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:48:46.773821 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.773803    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-token" pod "fdbd38f7-a556-4fdf-acfc-9077aac74ec7" (UID: "fdbd38f7-a556-4fdf-acfc-9077aac74ec7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:48:46.873842 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.873768    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.873842 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.873794    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jgcz\" (UniqueName: \"kubernetes.io/projected/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-kube-api-access-7jgcz\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.873842 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.873806    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.873842 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.873815    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.873842 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.873824    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdbd38f7-a556-4fdf-acfc-9077aac74ec7-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:46.896277 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.896240    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"]
Nov 02 23:48:46.899394 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:46.899365    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-2crzr"]
Nov 02 23:48:47.220873 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:47.220788    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"]
Nov 02 23:48:47.242534 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:47.242505    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ba1918-openshift-default-5f94dd79fc-sbvhr"]
Nov 02 23:48:47.709294 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:47.709265    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"
Nov 02 23:48:47.709294 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:47.709284    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5" event={"ID":"fdbd38f7-a556-4fdf-acfc-9077aac74ec7","Type":"ContainerDied","Data":"cb9fa0a2c94e51875df0d37b0f506f9cb9cac908a3fb0e4637736c662f5de710"}
Nov 02 23:48:47.709814 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:47.709339    4294 scope.go:122] "RemoveContainer" containerID="602bd9a14dc375db51bcdca7e2894637533b92e7ea7c6f2da44b005cf4b091cc"
Nov 02 23:48:47.869107 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:47.869075    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"]
Nov 02 23:48:47.914767 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:47.914733    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-zqbk5"]
Nov 02 23:48:48.411537 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:48.411508    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:48.548179 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:48.548148    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4571d9d7-7554-444c-a4d7-ba0daf1006fc" path="/var/lib/kubelet/pods/4571d9d7-7554-444c-a4d7-ba0daf1006fc/volumes"
Nov 02 23:48:48.548701 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:48.548680    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d078f228-be56-497c-87f6-8e94b080770c" path="/var/lib/kubelet/pods/d078f228-be56-497c-87f6-8e94b080770c/volumes"
Nov 02 23:48:48.549206 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:48.549187    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdbd38f7-a556-4fdf-acfc-9077aac74ec7" path="/var/lib/kubelet/pods/fdbd38f7-a556-4fdf-acfc-9077aac74ec7/volumes"
Nov 02 23:48:52.910787 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:52.910753    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:53.800817 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:53.800742    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:54.135750 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:54.135673    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:54.300927 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:54.300898    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:57.423468 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:57.423430    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:59.255880 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.255814    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh" podUID="94191650-b4f0-4fe2-ad32-cf98558fd894" containerName="main" containerID="cri-o://677ff13534b4142308b2bbdec1cd8b084dc7d05c85a3095bde0369ccc47eaee3" gracePeriod=45
Nov 02 23:48:59.304461 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.304427    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:48:59.525071 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.525007    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:48:59.613598 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.613572    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-tmp-dir\") pod \"94191650-b4f0-4fe2-ad32-cf98558fd894\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") "
Nov 02 23:48:59.613727 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.613605    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-dshm\" (UniqueName: \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-dshm\") pod \"94191650-b4f0-4fe2-ad32-cf98558fd894\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") "
Nov 02 23:48:59.613727 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.613683    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-model-cache\") pod \"94191650-b4f0-4fe2-ad32-cf98558fd894\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") "
Nov 02 23:48:59.613727 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.613721    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94191650-b4f0-4fe2-ad32-cf98558fd894-tls-certs\" (UniqueName: \"kubernetes.io/secret/94191650-b4f0-4fe2-ad32-cf98558fd894-tls-certs\") pod \"94191650-b4f0-4fe2-ad32-cf98558fd894\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") "
Nov 02 23:48:59.613860 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.613772    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94191650-b4f0-4fe2-ad32-cf98558fd894-kube-api-access-qjgj4\" (UniqueName: \"kubernetes.io/projected/94191650-b4f0-4fe2-ad32-cf98558fd894-kube-api-access-qjgj4\") pod \"94191650-b4f0-4fe2-ad32-cf98558fd894\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") "
Nov 02 23:48:59.613860 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.613806    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-home\" (UniqueName: \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-home\") pod \"94191650-b4f0-4fe2-ad32-cf98558fd894\" (UID: \"94191650-b4f0-4fe2-ad32-cf98558fd894\") "
Nov 02 23:48:59.613860 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.613806    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-tmp-dir" pod "94191650-b4f0-4fe2-ad32-cf98558fd894" (UID: "94191650-b4f0-4fe2-ad32-cf98558fd894"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:48:59.614055 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.613940    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-model-cache" pod "94191650-b4f0-4fe2-ad32-cf98558fd894" (UID: "94191650-b4f0-4fe2-ad32-cf98558fd894"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:48:59.614121 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.614067    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-home" pod "94191650-b4f0-4fe2-ad32-cf98558fd894" (UID: "94191650-b4f0-4fe2-ad32-cf98558fd894"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:48:59.614297 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.614274    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:59.614297 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.614295    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:59.614469 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.614305    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:59.615894 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.615868    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-dshm" pod "94191650-b4f0-4fe2-ad32-cf98558fd894" (UID: "94191650-b4f0-4fe2-ad32-cf98558fd894"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:48:59.616021 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.615935    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94191650-b4f0-4fe2-ad32-cf98558fd894-kube-api-access-qjgj4" pod "94191650-b4f0-4fe2-ad32-cf98558fd894" (UID: "94191650-b4f0-4fe2-ad32-cf98558fd894"). InnerVolumeSpecName "kube-api-access-qjgj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:48:59.616021 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.615942    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94191650-b4f0-4fe2-ad32-cf98558fd894-tls-certs" pod "94191650-b4f0-4fe2-ad32-cf98558fd894" (UID: "94191650-b4f0-4fe2-ad32-cf98558fd894"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:48:59.715674 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.715645    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjgj4\" (UniqueName: \"kubernetes.io/projected/94191650-b4f0-4fe2-ad32-cf98558fd894-kube-api-access-qjgj4\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:59.715674 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.715671    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94191650-b4f0-4fe2-ad32-cf98558fd894-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:59.715674 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.715680    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94191650-b4f0-4fe2-ad32-cf98558fd894-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:48:59.800144 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.800039    4294 generic.go:356] "Generic (PLEG): container finished" podID="94191650-b4f0-4fe2-ad32-cf98558fd894" containerID="677ff13534b4142308b2bbdec1cd8b084dc7d05c85a3095bde0369ccc47eaee3" exitCode=0
Nov 02 23:48:59.800338 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.800172    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh" event={"ID":"94191650-b4f0-4fe2-ad32-cf98558fd894","Type":"ContainerDied","Data":"677ff13534b4142308b2bbdec1cd8b084dc7d05c85a3095bde0369ccc47eaee3"}
Nov 02 23:48:59.800338 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.800209    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh" event={"ID":"94191650-b4f0-4fe2-ad32-cf98558fd894","Type":"ContainerDied","Data":"aa9b5c07cce41d765b313c0a6090327c2476c047e6141fbfe94c849286917728"}
Nov 02 23:48:59.800338 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.800242    4294 scope.go:122] "RemoveContainer" containerID="677ff13534b4142308b2bbdec1cd8b084dc7d05c85a3095bde0369ccc47eaee3"
Nov 02 23:48:59.800858 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.800439    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"
Nov 02 23:48:59.812134 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.812107    4294 scope.go:122] "RemoveContainer" containerID="677ff13534b4142308b2bbdec1cd8b084dc7d05c85a3095bde0369ccc47eaee3"
Nov 02 23:48:59.812438 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:48:59.812412    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"677ff13534b4142308b2bbdec1cd8b084dc7d05c85a3095bde0369ccc47eaee3\": container with ID starting with 677ff13534b4142308b2bbdec1cd8b084dc7d05c85a3095bde0369ccc47eaee3 not found: ID does not exist" containerID="677ff13534b4142308b2bbdec1cd8b084dc7d05c85a3095bde0369ccc47eaee3"
Nov 02 23:48:59.812532 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.812445    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"677ff13534b4142308b2bbdec1cd8b084dc7d05c85a3095bde0369ccc47eaee3"} err="failed to get container status \"677ff13534b4142308b2bbdec1cd8b084dc7d05c85a3095bde0369ccc47eaee3\": rpc error: code = NotFound desc = could not find container \"677ff13534b4142308b2bbdec1cd8b084dc7d05c85a3095bde0369ccc47eaee3\": container with ID starting with 677ff13534b4142308b2bbdec1cd8b084dc7d05c85a3095bde0369ccc47eaee3 not found: ID does not exist"
Nov 02 23:48:59.958648 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.958617    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"]
Nov 02 23:48:59.961469 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:48:59.961444    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-c5148c/auth-test-model-c5148c-kserve-6c96748cf7-h78vh"]
Nov 02 23:49:00.548556 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:00.548522    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94191650-b4f0-4fe2-ad32-cf98558fd894" path="/var/lib/kubelet/pods/94191650-b4f0-4fe2-ad32-cf98558fd894/volumes"
Nov 02 23:49:01.908913 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:01.908880    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:04.975112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:04.975078    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7421-da1746"]
Nov 02 23:49:04.975642 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:04.975628    4294 cpu_manager.go:414] "containerMap: removing container" podUID="d078f228-be56-497c-87f6-8e94b080770c" containerName="istio-proxy"
Nov 02 23:49:04.975686 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:04.975643    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d078f228-be56-497c-87f6-8e94b080770c" containerName="istio-proxy"
Nov 02 23:49:04.975686 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:04.975655    4294 cpu_manager.go:414] "containerMap: removing container" podUID="4571d9d7-7554-444c-a4d7-ba0daf1006fc" containerName="istio-proxy"
Nov 02 23:49:04.975686 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:04.975660    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4571d9d7-7554-444c-a4d7-ba0daf1006fc" containerName="istio-proxy"
Nov 02 23:49:04.975686 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:04.975667    4294 cpu_manager.go:414] "containerMap: removing container" podUID="fdbd38f7-a556-4fdf-acfc-9077aac74ec7" containerName="istio-proxy"
Nov 02 23:49:04.975686 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:04.975672    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdbd38f7-a556-4fdf-acfc-9077aac74ec7" containerName="istio-proxy"
Nov 02 23:49:04.975854 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:04.975689    4294 cpu_manager.go:414] "containerMap: removing container" podUID="4571d9d7-7554-444c-a4d7-ba0daf1006fc" containerName="istio-proxy"
Nov 02 23:49:04.975854 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:04.975694    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4571d9d7-7554-444c-a4d7-ba0daf1006fc" containerName="istio-proxy"
Nov 02 23:49:04.975854 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:04.975706    4294 cpu_manager.go:414] "containerMap: removing container" podUID="28664c33-54e8-4ced-9a16-dc46980a8bf5" containerName="payload-pre-processing"
Nov 02 23:49:04.975854 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:04.975711    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28664c33-54e8-4ced-9a16-dc46980a8bf5" containerName="payload-pre-processing"
Nov 02 23:49:04.975854 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:04.975738    4294 cpu_manager.go:414] "containerMap: removing container" podUID="94191650-b4f0-4fe2-ad32-cf98558fd894" containerName="main"
Nov 02 23:49:04.975854 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:04.975741    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94191650-b4f0-4fe2-ad32-cf98558fd894" containerName="main"
Nov 02 23:49:04.975854 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:04.975820    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="94191650-b4f0-4fe2-ad32-cf98558fd894" containerName="main"
Nov 02 23:49:04.975854 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:04.975830    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d078f228-be56-497c-87f6-8e94b080770c" containerName="istio-proxy"
Nov 02 23:49:04.975854 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:04.975835    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4571d9d7-7554-444c-a4d7-ba0daf1006fc" containerName="istio-proxy"
Nov 02 23:49:04.975854 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:04.975840    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdbd38f7-a556-4fdf-acfc-9077aac74ec7" containerName="istio-proxy"
Nov 02 23:49:04.975854 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:04.975848    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4571d9d7-7554-444c-a4d7-ba0daf1006fc" containerName="istio-proxy"
Nov 02 23:49:04.975854 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:04.975856    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="28664c33-54e8-4ced-9a16-dc46980a8bf5" containerName="payload-pre-processing"
Nov 02 23:49:04.982615 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:04.982583    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7421-da1746"
Nov 02 23:49:05.029996 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:05.029947    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7421-da1746"]
Nov 02 23:49:05.073104 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:05.073076    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbvzc\" (UniqueName: \"kubernetes.io/projected/3ca1586d-0a54-45f5-869c-84dfad5b4de7-kube-api-access-xbvzc\") pod \"test-curl-7421-da1746\" (UID: \"3ca1586d-0a54-45f5-869c-84dfad5b4de7\") " pod="opendatahub/test-curl-7421-da1746"
Nov 02 23:49:05.073917 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:05.073899    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wbwlb\""
Nov 02 23:49:05.174936 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:05.174902    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbvzc\" (UniqueName: \"kubernetes.io/projected/3ca1586d-0a54-45f5-869c-84dfad5b4de7-kube-api-access-xbvzc\") pod \"test-curl-7421-da1746\" (UID: \"3ca1586d-0a54-45f5-869c-84dfad5b4de7\") " pod="opendatahub/test-curl-7421-da1746"
Nov 02 23:49:05.229202 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:05.229138    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbvzc\" (UniqueName: \"kubernetes.io/projected/3ca1586d-0a54-45f5-869c-84dfad5b4de7-kube-api-access-xbvzc\") pod \"test-curl-7421-da1746\" (UID: \"3ca1586d-0a54-45f5-869c-84dfad5b4de7\") " pod="opendatahub/test-curl-7421-da1746"
Nov 02 23:49:05.296965 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:05.296939    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7421-da1746"
Nov 02 23:49:05.661751 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:49:05.661714    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ca1586d_0a54_45f5_869c_84dfad5b4de7.slice/crio-317f9fddb37c88e96e7910c1808cae7cf931b87efafc192a2bcd406f80108d27 WatchSource:0}: Error finding container 317f9fddb37c88e96e7910c1808cae7cf931b87efafc192a2bcd406f80108d27: Status 404 returned error can't find the container with id 317f9fddb37c88e96e7910c1808cae7cf931b87efafc192a2bcd406f80108d27
Nov 02 23:49:05.661751 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:05.661724    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7421-da1746"]
Nov 02 23:49:05.663600 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:05.663584    4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 23:49:05.836952 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:05.836918    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7421-da1746" event={"ID":"3ca1586d-0a54-45f5-869c-84dfad5b4de7","Type":"ContainerStarted","Data":"317f9fddb37c88e96e7910c1808cae7cf931b87efafc192a2bcd406f80108d27"}
Nov 02 23:49:06.401814 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:06.401753    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:07.850655 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:07.850616    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7421-da1746" event={"ID":"3ca1586d-0a54-45f5-869c-84dfad5b4de7","Type":"ContainerStarted","Data":"a39c5b9919a85adc57c5ef11bd2e2849ce4077ccc7a7eec04d273e5867b44c73"}
Nov 02 23:49:07.958440 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:07.958382    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7421-da1746" podStartSLOduration=2.679188312 podStartE2EDuration="3.958366821s" totalImagesPullingTime="1.279178509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:49:07.957252167 +0000 UTC m=+3387.525606954" watchObservedRunningTime="2026-11-02 23:49:07.958366821 +0000 UTC m=+3387.526721610"
Nov 02 23:49:09.494098 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:09.494055    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7421-da1746"]
Nov 02 23:49:09.497678 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:09.497652    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7421-da1746"]
Nov 02 23:49:09.817412 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:09.817333    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:09.863394 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:09.863339    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7421-da1746" podUID="3ca1586d-0a54-45f5-869c-84dfad5b4de7" containerName="test-curl-7421-da1746" containerID="cri-o://a39c5b9919a85adc57c5ef11bd2e2849ce4077ccc7a7eec04d273e5867b44c73" gracePeriod=2
Nov 02 23:49:09.907417 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:09.907349    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7421-138964"]
Nov 02 23:49:09.907997 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:09.907968    4294 cpu_manager.go:414] "containerMap: removing container" podUID="3ca1586d-0a54-45f5-869c-84dfad5b4de7" containerName="test-curl-7421-da1746"
Nov 02 23:49:09.908052 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:09.907998    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ca1586d-0a54-45f5-869c-84dfad5b4de7" containerName="test-curl-7421-da1746"
Nov 02 23:49:09.908133 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:09.908123    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ca1586d-0a54-45f5-869c-84dfad5b4de7" containerName="test-curl-7421-da1746"
Nov 02 23:49:09.911923 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:09.911902    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7421-138964"
Nov 02 23:49:09.912320 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:49:09.912294    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7421-da1746\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="3ca1586d-0a54-45f5-869c-84dfad5b4de7" pod="opendatahub/test-curl-7421-da1746"
Nov 02 23:49:09.961397 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:09.961360    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7421-138964"]
Nov 02 23:49:10.030603 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:10.030568    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99jlz\" (UniqueName: \"kubernetes.io/projected/66e98a63-2790-4dce-b3c4-7b0683b4835d-kube-api-access-99jlz\") pod \"test-curl-7421-138964\" (UID: \"66e98a63-2790-4dce-b3c4-7b0683b4835d\") " pod="opendatahub/test-curl-7421-138964"
Nov 02 23:49:10.066241 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:49:10.066203    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7421-da1746\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="3ca1586d-0a54-45f5-869c-84dfad5b4de7" pod="opendatahub/test-curl-7421-da1746"
Nov 02 23:49:10.131836 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:10.131758    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99jlz\" (UniqueName: \"kubernetes.io/projected/66e98a63-2790-4dce-b3c4-7b0683b4835d-kube-api-access-99jlz\") pod \"test-curl-7421-138964\" (UID: \"66e98a63-2790-4dce-b3c4-7b0683b4835d\") " pod="opendatahub/test-curl-7421-138964"
Nov 02 23:49:10.186291 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:10.186261    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99jlz\" (UniqueName: \"kubernetes.io/projected/66e98a63-2790-4dce-b3c4-7b0683b4835d-kube-api-access-99jlz\") pod \"test-curl-7421-138964\" (UID: \"66e98a63-2790-4dce-b3c4-7b0683b4835d\") " pod="opendatahub/test-curl-7421-138964"
Nov 02 23:49:10.234954 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:10.234928    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7421-138964"
Nov 02 23:49:10.593099 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:49:10.593064    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7421-da1746\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="3ca1586d-0a54-45f5-869c-84dfad5b4de7" pod="opendatahub/test-curl-7421-da1746"
Nov 02 23:49:10.598897 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:10.598870    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7421-138964"]
Nov 02 23:49:10.599097 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:49:10.599074    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66e98a63_2790_4dce_b3c4_7b0683b4835d.slice/crio-809de0d3290b8f3ac0acc1d0daf233639fca99e4c4c8b7e628633482c8f1abb7 WatchSource:0}: Error finding container 809de0d3290b8f3ac0acc1d0daf233639fca99e4c4c8b7e628633482c8f1abb7: Status 404 returned error can't find the container with id 809de0d3290b8f3ac0acc1d0daf233639fca99e4c4c8b7e628633482c8f1abb7
Nov 02 23:49:10.768798 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:10.768759    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:10.874119 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:10.874033    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7421-138964" event={"ID":"66e98a63-2790-4dce-b3c4-7b0683b4835d","Type":"ContainerStarted","Data":"7d20880b16d676b9e50d396945de99f130a1e8aa099a34c8d75a493767be62ef"}
Nov 02 23:49:10.874119 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:10.874085    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7421-138964" event={"ID":"66e98a63-2790-4dce-b3c4-7b0683b4835d","Type":"ContainerStarted","Data":"809de0d3290b8f3ac0acc1d0daf233639fca99e4c4c8b7e628633482c8f1abb7"}
Nov 02 23:49:10.946304 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:10.946273    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:10.984128 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:10.984064    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7421-138964" podStartSLOduration=1.984049972 podStartE2EDuration="1.984049972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:49:10.981171278 +0000 UTC m=+3390.549526063" watchObservedRunningTime="2026-11-02 23:49:10.984049972 +0000 UTC m=+3390.552404759"
Nov 02 23:49:11.761702 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:49:11.761671    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:49:12.107968 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.107946    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7421-da1746"
Nov 02 23:49:12.155813 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.155783    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ca1586d-0a54-45f5-869c-84dfad5b4de7-kube-api-access-xbvzc\" (UniqueName: \"kubernetes.io/projected/3ca1586d-0a54-45f5-869c-84dfad5b4de7-kube-api-access-xbvzc\") pod \"3ca1586d-0a54-45f5-869c-84dfad5b4de7\" (UID: \"3ca1586d-0a54-45f5-869c-84dfad5b4de7\") "
Nov 02 23:49:12.157655 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.157630    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca1586d-0a54-45f5-869c-84dfad5b4de7-kube-api-access-xbvzc" pod "3ca1586d-0a54-45f5-869c-84dfad5b4de7" (UID: "3ca1586d-0a54-45f5-869c-84dfad5b4de7"). InnerVolumeSpecName "kube-api-access-xbvzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:49:12.257153 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.257120    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbvzc\" (UniqueName: \"kubernetes.io/projected/3ca1586d-0a54-45f5-869c-84dfad5b4de7-kube-api-access-xbvzc\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:49:12.360545 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.360202    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7421-138964"]
Nov 02 23:49:12.364420 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.364392    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7421-138964"]
Nov 02 23:49:12.499091 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.499056    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7421-52631f"]
Nov 02 23:49:12.499685 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.499670    4294 cpu_manager.go:414] "containerMap: removing container" podUID="66e98a63-2790-4dce-b3c4-7b0683b4835d" containerName="test-curl-7421-138964"
Nov 02 23:49:12.499731 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.499686    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e98a63-2790-4dce-b3c4-7b0683b4835d" containerName="test-curl-7421-138964"
Nov 02 23:49:12.499782 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.499773    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e98a63-2790-4dce-b3c4-7b0683b4835d" containerName="test-curl-7421-138964"
Nov 02 23:49:12.503233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.503218    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7421-52631f"
Nov 02 23:49:12.547900 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.547861    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ca1586d-0a54-45f5-869c-84dfad5b4de7" path="/var/lib/kubelet/pods/3ca1586d-0a54-45f5-869c-84dfad5b4de7/volumes"
Nov 02 23:49:12.553594 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.553562    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7421-52631f"]
Nov 02 23:49:12.562005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.560862    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qw76\" (UniqueName: \"kubernetes.io/projected/551d6725-54ab-4d8e-ad12-746ab4c929f3-kube-api-access-5qw76\") pod \"test-curl-7421-52631f\" (UID: \"551d6725-54ab-4d8e-ad12-746ab4c929f3\") " pod="opendatahub/test-curl-7421-52631f"
Nov 02 23:49:12.661951 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.661871    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qw76\" (UniqueName: \"kubernetes.io/projected/551d6725-54ab-4d8e-ad12-746ab4c929f3-kube-api-access-5qw76\") pod \"test-curl-7421-52631f\" (UID: \"551d6725-54ab-4d8e-ad12-746ab4c929f3\") " pod="opendatahub/test-curl-7421-52631f"
Nov 02 23:49:12.716607 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.716582    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qw76\" (UniqueName: \"kubernetes.io/projected/551d6725-54ab-4d8e-ad12-746ab4c929f3-kube-api-access-5qw76\") pod \"test-curl-7421-52631f\" (UID: \"551d6725-54ab-4d8e-ad12-746ab4c929f3\") " pod="opendatahub/test-curl-7421-52631f"
Nov 02 23:49:12.814958 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.814921    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7421-52631f"
Nov 02 23:49:12.892245 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.892214    4294 generic.go:356] "Generic (PLEG): container finished" podID="3ca1586d-0a54-45f5-869c-84dfad5b4de7" containerID="a39c5b9919a85adc57c5ef11bd2e2849ce4077ccc7a7eec04d273e5867b44c73" exitCode=137
Nov 02 23:49:12.892412 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.892280    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7421-da1746"
Nov 02 23:49:12.892486 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.892431    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7421-138964" podUID="66e98a63-2790-4dce-b3c4-7b0683b4835d" containerName="test-curl-7421-138964" containerID="cri-o://7d20880b16d676b9e50d396945de99f130a1e8aa099a34c8d75a493767be62ef" gracePeriod=2
Nov 02 23:49:12.892785 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.892574    4294 scope.go:122] "RemoveContainer" containerID="a39c5b9919a85adc57c5ef11bd2e2849ce4077ccc7a7eec04d273e5867b44c73"
Nov 02 23:49:12.904405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.904382    4294 scope.go:122] "RemoveContainer" containerID="a39c5b9919a85adc57c5ef11bd2e2849ce4077ccc7a7eec04d273e5867b44c73"
Nov 02 23:49:12.904771 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:49:12.904747    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a39c5b9919a85adc57c5ef11bd2e2849ce4077ccc7a7eec04d273e5867b44c73\": container with ID starting with a39c5b9919a85adc57c5ef11bd2e2849ce4077ccc7a7eec04d273e5867b44c73 not found: ID does not exist" containerID="a39c5b9919a85adc57c5ef11bd2e2849ce4077ccc7a7eec04d273e5867b44c73"
Nov 02 23:49:12.904869 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:12.904780    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a39c5b9919a85adc57c5ef11bd2e2849ce4077ccc7a7eec04d273e5867b44c73"} err="failed to get container status \"a39c5b9919a85adc57c5ef11bd2e2849ce4077ccc7a7eec04d273e5867b44c73\": rpc error: code = NotFound desc = could not find container \"a39c5b9919a85adc57c5ef11bd2e2849ce4077ccc7a7eec04d273e5867b44c73\": container with ID starting with a39c5b9919a85adc57c5ef11bd2e2849ce4077ccc7a7eec04d273e5867b44c73 not found: ID does not exist"
Nov 02 23:49:13.181073 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:13.181043    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7421-52631f"]
Nov 02 23:49:13.181188 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:49:13.181138    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod551d6725_54ab_4d8e_ad12_746ab4c929f3.slice/crio-ed450f11c82c2145afcc6910e1280dd2b14fc00f599f8953a924a211d11d3f45 WatchSource:0}: Error finding container ed450f11c82c2145afcc6910e1280dd2b14fc00f599f8953a924a211d11d3f45: Status 404 returned error can't find the container with id ed450f11c82c2145afcc6910e1280dd2b14fc00f599f8953a924a211d11d3f45
Nov 02 23:49:13.901108 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:13.901074    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7421-52631f" event={"ID":"551d6725-54ab-4d8e-ad12-746ab4c929f3","Type":"ContainerStarted","Data":"7f35f48c6ad5e6edbf25418ad7a3153be76213528ac0cf1cbdbe461e02638162"}
Nov 02 23:49:13.901108 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:13.901115    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7421-52631f" event={"ID":"551d6725-54ab-4d8e-ad12-746ab4c929f3","Type":"ContainerStarted","Data":"ed450f11c82c2145afcc6910e1280dd2b14fc00f599f8953a924a211d11d3f45"}
Nov 02 23:49:14.010525 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:14.010470    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7421-52631f" podStartSLOduration=2.010456181 podStartE2EDuration="2.010456181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:49:14.007421383 +0000 UTC m=+3393.575776171" watchObservedRunningTime="2026-11-02 23:49:14.010456181 +0000 UTC m=+3393.578810967"
Nov 02 23:49:15.141890 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.141863    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7421-138964"
Nov 02 23:49:15.188121 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.188046    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e98a63-2790-4dce-b3c4-7b0683b4835d-kube-api-access-99jlz\" (UniqueName: \"kubernetes.io/projected/66e98a63-2790-4dce-b3c4-7b0683b4835d-kube-api-access-99jlz\") pod \"66e98a63-2790-4dce-b3c4-7b0683b4835d\" (UID: \"66e98a63-2790-4dce-b3c4-7b0683b4835d\") "
Nov 02 23:49:15.190118 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.190092    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e98a63-2790-4dce-b3c4-7b0683b4835d-kube-api-access-99jlz" pod "66e98a63-2790-4dce-b3c4-7b0683b4835d" (UID: "66e98a63-2790-4dce-b3c4-7b0683b4835d"). InnerVolumeSpecName "kube-api-access-99jlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:49:15.289693 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.289654    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99jlz\" (UniqueName: \"kubernetes.io/projected/66e98a63-2790-4dce-b3c4-7b0683b4835d-kube-api-access-99jlz\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:49:15.486038 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.485939    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7421-52631f"]
Nov 02 23:49:15.491878 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.491844    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7421-52631f"]
Nov 02 23:49:15.556178 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.556147    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:15.622833 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.622801    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7421-7d08fc"]
Nov 02 23:49:15.623440 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.623425    4294 cpu_manager.go:414] "containerMap: removing container" podUID="551d6725-54ab-4d8e-ad12-746ab4c929f3" containerName="test-curl-7421-52631f"
Nov 02 23:49:15.623494 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.623441    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551d6725-54ab-4d8e-ad12-746ab4c929f3" containerName="test-curl-7421-52631f"
Nov 02 23:49:15.623555 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.623545    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="551d6725-54ab-4d8e-ad12-746ab4c929f3" containerName="test-curl-7421-52631f"
Nov 02 23:49:15.626912 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.626897    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7421-7d08fc"
Nov 02 23:49:15.676544 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.676512    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7421-7d08fc"]
Nov 02 23:49:15.694926 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.694887    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6d9d\" (UniqueName: \"kubernetes.io/projected/5ae93e40-715a-4c89-97e2-bd9f3e7eb400-kube-api-access-d6d9d\") pod \"test-curl-7421-7d08fc\" (UID: \"5ae93e40-715a-4c89-97e2-bd9f3e7eb400\") " pod="opendatahub/test-curl-7421-7d08fc"
Nov 02 23:49:15.796484 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.796445    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6d9d\" (UniqueName: \"kubernetes.io/projected/5ae93e40-715a-4c89-97e2-bd9f3e7eb400-kube-api-access-d6d9d\") pod \"test-curl-7421-7d08fc\" (UID: \"5ae93e40-715a-4c89-97e2-bd9f3e7eb400\") " pod="opendatahub/test-curl-7421-7d08fc"
Nov 02 23:49:15.850682 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.850646    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6d9d\" (UniqueName: \"kubernetes.io/projected/5ae93e40-715a-4c89-97e2-bd9f3e7eb400-kube-api-access-d6d9d\") pod \"test-curl-7421-7d08fc\" (UID: \"5ae93e40-715a-4c89-97e2-bd9f3e7eb400\") " pod="opendatahub/test-curl-7421-7d08fc"
Nov 02 23:49:15.918478 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.918435    4294 generic.go:356] "Generic (PLEG): container finished" podID="66e98a63-2790-4dce-b3c4-7b0683b4835d" containerID="7d20880b16d676b9e50d396945de99f130a1e8aa099a34c8d75a493767be62ef" exitCode=137
Nov 02 23:49:15.918762 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.918503    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7421-138964"
Nov 02 23:49:15.919019 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.918974    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7421-52631f" podUID="551d6725-54ab-4d8e-ad12-746ab4c929f3" containerName="test-curl-7421-52631f" containerID="cri-o://7f35f48c6ad5e6edbf25418ad7a3153be76213528ac0cf1cbdbe461e02638162" gracePeriod=2
Nov 02 23:49:15.919320 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.918534    4294 scope.go:122] "RemoveContainer" containerID="7d20880b16d676b9e50d396945de99f130a1e8aa099a34c8d75a493767be62ef"
Nov 02 23:49:15.936818 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.936527    4294 scope.go:122] "RemoveContainer" containerID="7d20880b16d676b9e50d396945de99f130a1e8aa099a34c8d75a493767be62ef"
Nov 02 23:49:15.937421 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:49:15.937386    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d20880b16d676b9e50d396945de99f130a1e8aa099a34c8d75a493767be62ef\": container with ID starting with 7d20880b16d676b9e50d396945de99f130a1e8aa099a34c8d75a493767be62ef not found: ID does not exist" containerID="7d20880b16d676b9e50d396945de99f130a1e8aa099a34c8d75a493767be62ef"
Nov 02 23:49:15.937517 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.937433    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d20880b16d676b9e50d396945de99f130a1e8aa099a34c8d75a493767be62ef"} err="failed to get container status \"7d20880b16d676b9e50d396945de99f130a1e8aa099a34c8d75a493767be62ef\": rpc error: code = NotFound desc = could not find container \"7d20880b16d676b9e50d396945de99f130a1e8aa099a34c8d75a493767be62ef\": container with ID starting with 7d20880b16d676b9e50d396945de99f130a1e8aa099a34c8d75a493767be62ef not found: ID does not exist"
Nov 02 23:49:15.941775 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:15.941745    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7421-7d08fc"
Nov 02 23:49:15.968173 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:49:15.968139    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7421-52631f\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="551d6725-54ab-4d8e-ad12-746ab4c929f3" pod="opendatahub/test-curl-7421-52631f"
Nov 02 23:49:16.017083 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:49:16.017030    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7421-52631f\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="551d6725-54ab-4d8e-ad12-746ab4c929f3" pod="opendatahub/test-curl-7421-52631f"
Nov 02 23:49:16.065769 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:49:16.065690    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7421-138964\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="66e98a63-2790-4dce-b3c4-7b0683b4835d" pod="opendatahub/test-curl-7421-138964"
Nov 02 23:49:16.323902 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:49:16.323867    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ae93e40_715a_4c89_97e2_bd9f3e7eb400.slice/crio-778f4e663345cbe39c32b654c482f4d1134c6f4004eb8fa47ec5e49754195ed1 WatchSource:0}: Error finding container 778f4e663345cbe39c32b654c482f4d1134c6f4004eb8fa47ec5e49754195ed1: Status 404 returned error can't find the container with id 778f4e663345cbe39c32b654c482f4d1134c6f4004eb8fa47ec5e49754195ed1
Nov 02 23:49:16.324512 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:16.324484    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7421-7d08fc"]
Nov 02 23:49:16.549212 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:16.549182    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66e98a63-2790-4dce-b3c4-7b0683b4835d" path="/var/lib/kubelet/pods/66e98a63-2790-4dce-b3c4-7b0683b4835d/volumes"
Nov 02 23:49:16.926778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:16.926734    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7421-7d08fc" event={"ID":"5ae93e40-715a-4c89-97e2-bd9f3e7eb400","Type":"ContainerStarted","Data":"1cda5575929661aaa5f2637a1fbc001b1276ed64c9aed91bc4a3e6856e803328"}
Nov 02 23:49:16.926778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:16.926784    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7421-7d08fc" event={"ID":"5ae93e40-715a-4c89-97e2-bd9f3e7eb400","Type":"ContainerStarted","Data":"778f4e663345cbe39c32b654c482f4d1134c6f4004eb8fa47ec5e49754195ed1"}
Nov 02 23:49:17.035772 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:17.035721    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7421-7d08fc" podStartSLOduration=2.03570727 podStartE2EDuration="2.03570727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:49:17.034129167 +0000 UTC m=+3396.602483954" watchObservedRunningTime="2026-11-02 23:49:17.03570727 +0000 UTC m=+3396.604062056"
Nov 02 23:49:18.157247 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.157217    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7421-52631f"
Nov 02 23:49:18.227912 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.227834    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551d6725-54ab-4d8e-ad12-746ab4c929f3-kube-api-access-5qw76\" (UniqueName: \"kubernetes.io/projected/551d6725-54ab-4d8e-ad12-746ab4c929f3-kube-api-access-5qw76\") pod \"551d6725-54ab-4d8e-ad12-746ab4c929f3\" (UID: \"551d6725-54ab-4d8e-ad12-746ab4c929f3\") "
Nov 02 23:49:18.229766 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.229741    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551d6725-54ab-4d8e-ad12-746ab4c929f3-kube-api-access-5qw76" pod "551d6725-54ab-4d8e-ad12-746ab4c929f3" (UID: "551d6725-54ab-4d8e-ad12-746ab4c929f3"). InnerVolumeSpecName "kube-api-access-5qw76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:49:18.329481 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.329450    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qw76\" (UniqueName: \"kubernetes.io/projected/551d6725-54ab-4d8e-ad12-746ab4c929f3-kube-api-access-5qw76\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:49:18.410511 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.410422    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7421-7d08fc"]
Nov 02 23:49:18.415317 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.415284    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7421-7d08fc"]
Nov 02 23:49:18.547526 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.547497    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551d6725-54ab-4d8e-ad12-746ab4c929f3" path="/var/lib/kubelet/pods/551d6725-54ab-4d8e-ad12-746ab4c929f3/volumes"
Nov 02 23:49:18.554350 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.554330    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7421-fc6765"]
Nov 02 23:49:18.554929 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.554910    4294 cpu_manager.go:414] "containerMap: removing container" podUID="5ae93e40-715a-4c89-97e2-bd9f3e7eb400" containerName="test-curl-7421-7d08fc"
Nov 02 23:49:18.554929 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.554926    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae93e40-715a-4c89-97e2-bd9f3e7eb400" containerName="test-curl-7421-7d08fc"
Nov 02 23:49:18.555103 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.555076    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae93e40-715a-4c89-97e2-bd9f3e7eb400" containerName="test-curl-7421-7d08fc"
Nov 02 23:49:18.558699 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.558685    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7421-fc6765"
Nov 02 23:49:18.608807 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.608781    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7421-fc6765"]
Nov 02 23:49:18.633409 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.633377    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v98cj\" (UniqueName: \"kubernetes.io/projected/1809bac7-6631-4311-a474-758f38821e75-kube-api-access-v98cj\") pod \"test-curl-7421-fc6765\" (UID: \"1809bac7-6631-4311-a474-758f38821e75\") " pod="opendatahub/test-curl-7421-fc6765"
Nov 02 23:49:18.734719 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.734686    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v98cj\" (UniqueName: \"kubernetes.io/projected/1809bac7-6631-4311-a474-758f38821e75-kube-api-access-v98cj\") pod \"test-curl-7421-fc6765\" (UID: \"1809bac7-6631-4311-a474-758f38821e75\") " pod="opendatahub/test-curl-7421-fc6765"
Nov 02 23:49:18.788613 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.788589    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v98cj\" (UniqueName: \"kubernetes.io/projected/1809bac7-6631-4311-a474-758f38821e75-kube-api-access-v98cj\") pod \"test-curl-7421-fc6765\" (UID: \"1809bac7-6631-4311-a474-758f38821e75\") " pod="opendatahub/test-curl-7421-fc6765"
Nov 02 23:49:18.871075 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.871006    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7421-fc6765"
Nov 02 23:49:18.945435 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.945401    4294 generic.go:356] "Generic (PLEG): container finished" podID="551d6725-54ab-4d8e-ad12-746ab4c929f3" containerID="7f35f48c6ad5e6edbf25418ad7a3153be76213528ac0cf1cbdbe461e02638162" exitCode=137
Nov 02 23:49:18.945715 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.945691    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7421-7d08fc" podUID="5ae93e40-715a-4c89-97e2-bd9f3e7eb400" containerName="test-curl-7421-7d08fc" containerID="cri-o://1cda5575929661aaa5f2637a1fbc001b1276ed64c9aed91bc4a3e6856e803328" gracePeriod=2
Nov 02 23:49:18.946118 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.946091    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7421-52631f"
Nov 02 23:49:18.946218 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.946160    4294 scope.go:122] "RemoveContainer" containerID="7f35f48c6ad5e6edbf25418ad7a3153be76213528ac0cf1cbdbe461e02638162"
Nov 02 23:49:18.955658 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.955635    4294 scope.go:122] "RemoveContainer" containerID="7f35f48c6ad5e6edbf25418ad7a3153be76213528ac0cf1cbdbe461e02638162"
Nov 02 23:49:18.955958 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:49:18.955940    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f35f48c6ad5e6edbf25418ad7a3153be76213528ac0cf1cbdbe461e02638162\": container with ID starting with 7f35f48c6ad5e6edbf25418ad7a3153be76213528ac0cf1cbdbe461e02638162 not found: ID does not exist" containerID="7f35f48c6ad5e6edbf25418ad7a3153be76213528ac0cf1cbdbe461e02638162"
Nov 02 23:49:18.956053 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:18.955964    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f35f48c6ad5e6edbf25418ad7a3153be76213528ac0cf1cbdbe461e02638162"} err="failed to get container status \"7f35f48c6ad5e6edbf25418ad7a3153be76213528ac0cf1cbdbe461e02638162\": rpc error: code = NotFound desc = could not find container \"7f35f48c6ad5e6edbf25418ad7a3153be76213528ac0cf1cbdbe461e02638162\": container with ID starting with 7f35f48c6ad5e6edbf25418ad7a3153be76213528ac0cf1cbdbe461e02638162 not found: ID does not exist"
Nov 02 23:49:19.233522 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:19.233496    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7421-fc6765"]
Nov 02 23:49:19.233841 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:49:19.233569    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1809bac7_6631_4311_a474_758f38821e75.slice/crio-588541fc6bc1edf4aabaa4882df8163c198f176275b7e98e6f87a8e43417d80b WatchSource:0}: Error finding container 588541fc6bc1edf4aabaa4882df8163c198f176275b7e98e6f87a8e43417d80b: Status 404 returned error can't find the container with id 588541fc6bc1edf4aabaa4882df8163c198f176275b7e98e6f87a8e43417d80b
Nov 02 23:49:19.953600 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:19.953568    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7421-fc6765" event={"ID":"1809bac7-6631-4311-a474-758f38821e75","Type":"ContainerStarted","Data":"2be044e93ccc2ddcfe1b5eb86ff7c62d08b50f6c83cab612e34884fcf63b6ff2"}
Nov 02 23:49:19.953600 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:19.953610    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7421-fc6765" event={"ID":"1809bac7-6631-4311-a474-758f38821e75","Type":"ContainerStarted","Data":"588541fc6bc1edf4aabaa4882df8163c198f176275b7e98e6f87a8e43417d80b"}
Nov 02 23:49:20.063306 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:20.063258    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7421-fc6765" podStartSLOduration=2.06324513 podStartE2EDuration="2.06324513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:49:20.061261234 +0000 UTC m=+3399.629616021" watchObservedRunningTime="2026-11-02 23:49:20.06324513 +0000 UTC m=+3399.631599916"
Nov 02 23:49:20.161862 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:20.161829    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:21.193378 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:21.193356    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7421-7d08fc"
Nov 02 23:49:21.261537 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:21.261514    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae93e40-715a-4c89-97e2-bd9f3e7eb400-kube-api-access-d6d9d\" (UniqueName: \"kubernetes.io/projected/5ae93e40-715a-4c89-97e2-bd9f3e7eb400-kube-api-access-d6d9d\") pod \"5ae93e40-715a-4c89-97e2-bd9f3e7eb400\" (UID: \"5ae93e40-715a-4c89-97e2-bd9f3e7eb400\") "
Nov 02 23:49:21.263373 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:21.263332    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae93e40-715a-4c89-97e2-bd9f3e7eb400-kube-api-access-d6d9d" pod "5ae93e40-715a-4c89-97e2-bd9f3e7eb400" (UID: "5ae93e40-715a-4c89-97e2-bd9f3e7eb400"). InnerVolumeSpecName "kube-api-access-d6d9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:49:21.363064 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:21.363039    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6d9d\" (UniqueName: \"kubernetes.io/projected/5ae93e40-715a-4c89-97e2-bd9f3e7eb400-kube-api-access-d6d9d\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:49:21.549546 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:21.549462    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7421-fc6765"]
Nov 02 23:49:21.552488 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:21.552460    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7421-fc6765"]
Nov 02 23:49:21.967085 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:21.966973    4294 generic.go:356] "Generic (PLEG): container finished" podID="5ae93e40-715a-4c89-97e2-bd9f3e7eb400" containerID="1cda5575929661aaa5f2637a1fbc001b1276ed64c9aed91bc4a3e6856e803328" exitCode=137
Nov 02 23:49:21.967085 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:21.967041    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7421-7d08fc"
Nov 02 23:49:21.967258 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:21.967090    4294 scope.go:122] "RemoveContainer" containerID="1cda5575929661aaa5f2637a1fbc001b1276ed64c9aed91bc4a3e6856e803328"
Nov 02 23:49:21.967315 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:21.967291    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7421-fc6765" podUID="1809bac7-6631-4311-a474-758f38821e75" containerName="test-curl-7421-fc6765" containerID="cri-o://2be044e93ccc2ddcfe1b5eb86ff7c62d08b50f6c83cab612e34884fcf63b6ff2" gracePeriod=2
Nov 02 23:49:21.982015 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:21.981992    4294 scope.go:122] "RemoveContainer" containerID="1cda5575929661aaa5f2637a1fbc001b1276ed64c9aed91bc4a3e6856e803328"
Nov 02 23:49:21.983087 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:49:21.983058    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cda5575929661aaa5f2637a1fbc001b1276ed64c9aed91bc4a3e6856e803328\": container with ID starting with 1cda5575929661aaa5f2637a1fbc001b1276ed64c9aed91bc4a3e6856e803328 not found: ID does not exist" containerID="1cda5575929661aaa5f2637a1fbc001b1276ed64c9aed91bc4a3e6856e803328"
Nov 02 23:49:21.983193 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:21.983096    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cda5575929661aaa5f2637a1fbc001b1276ed64c9aed91bc4a3e6856e803328"} err="failed to get container status \"1cda5575929661aaa5f2637a1fbc001b1276ed64c9aed91bc4a3e6856e803328\": rpc error: code = NotFound desc = could not find container \"1cda5575929661aaa5f2637a1fbc001b1276ed64c9aed91bc4a3e6856e803328\": container with ID starting with 1cda5575929661aaa5f2637a1fbc001b1276ed64c9aed91bc4a3e6856e803328 not found: ID does not exist"
Nov 02 23:49:22.016342 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:49:22.016304    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7421-fc6765\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="1809bac7-6631-4311-a474-758f38821e75" pod="opendatahub/test-curl-7421-fc6765"
Nov 02 23:49:22.065407 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:49:22.065381    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7421-7d08fc\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="5ae93e40-715a-4c89-97e2-bd9f3e7eb400" pod="opendatahub/test-curl-7421-7d08fc"
Nov 02 23:49:22.114064 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:49:22.114035    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7421-fc6765\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="1809bac7-6631-4311-a474-758f38821e75" pod="opendatahub/test-curl-7421-fc6765"
Nov 02 23:49:22.548046 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:22.548015    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae93e40-715a-4c89-97e2-bd9f3e7eb400" path="/var/lib/kubelet/pods/5ae93e40-715a-4c89-97e2-bd9f3e7eb400/volumes"
Nov 02 23:49:23.036856 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:23.036821    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:23.636722 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:23.636691    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:23.927767 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:23.927686    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:24.206909 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:24.206886    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7421-fc6765"
Nov 02 23:49:24.296457 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:24.296421    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1809bac7-6631-4311-a474-758f38821e75-kube-api-access-v98cj\" (UniqueName: \"kubernetes.io/projected/1809bac7-6631-4311-a474-758f38821e75-kube-api-access-v98cj\") pod \"1809bac7-6631-4311-a474-758f38821e75\" (UID: \"1809bac7-6631-4311-a474-758f38821e75\") "
Nov 02 23:49:24.298496 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:24.298468    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1809bac7-6631-4311-a474-758f38821e75-kube-api-access-v98cj" pod "1809bac7-6631-4311-a474-758f38821e75" (UID: "1809bac7-6631-4311-a474-758f38821e75"). InnerVolumeSpecName "kube-api-access-v98cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:49:24.397968 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:24.397937    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v98cj\" (UniqueName: \"kubernetes.io/projected/1809bac7-6631-4311-a474-758f38821e75-kube-api-access-v98cj\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:49:24.548528 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:24.548498    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1809bac7-6631-4311-a474-758f38821e75" path="/var/lib/kubelet/pods/1809bac7-6631-4311-a474-758f38821e75/volumes"
Nov 02 23:49:24.760523 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:24.760492    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:24.988510 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:24.988432    4294 generic.go:356] "Generic (PLEG): container finished" podID="1809bac7-6631-4311-a474-758f38821e75" containerID="2be044e93ccc2ddcfe1b5eb86ff7c62d08b50f6c83cab612e34884fcf63b6ff2" exitCode=137
Nov 02 23:49:24.988510 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:24.988484    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7421-fc6765"
Nov 02 23:49:24.988737 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:24.988532    4294 scope.go:122] "RemoveContainer" containerID="2be044e93ccc2ddcfe1b5eb86ff7c62d08b50f6c83cab612e34884fcf63b6ff2"
Nov 02 23:49:24.999026 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:24.999005    4294 scope.go:122] "RemoveContainer" containerID="2be044e93ccc2ddcfe1b5eb86ff7c62d08b50f6c83cab612e34884fcf63b6ff2"
Nov 02 23:49:24.999549 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:49:24.999518    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be044e93ccc2ddcfe1b5eb86ff7c62d08b50f6c83cab612e34884fcf63b6ff2\": container with ID starting with 2be044e93ccc2ddcfe1b5eb86ff7c62d08b50f6c83cab612e34884fcf63b6ff2 not found: ID does not exist" containerID="2be044e93ccc2ddcfe1b5eb86ff7c62d08b50f6c83cab612e34884fcf63b6ff2"
Nov 02 23:49:24.999678 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:24.999554    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be044e93ccc2ddcfe1b5eb86ff7c62d08b50f6c83cab612e34884fcf63b6ff2"} err="failed to get container status \"2be044e93ccc2ddcfe1b5eb86ff7c62d08b50f6c83cab612e34884fcf63b6ff2\": rpc error: code = NotFound desc = could not find container \"2be044e93ccc2ddcfe1b5eb86ff7c62d08b50f6c83cab612e34884fcf63b6ff2\": container with ID starting with 2be044e93ccc2ddcfe1b5eb86ff7c62d08b50f6c83cab612e34884fcf63b6ff2 not found: ID does not exist"
Nov 02 23:49:27.707796 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:27.707764    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:28.030647 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:28.030618    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:28.261781 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:28.261752    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:29.631903 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:29.631874    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:30.859739 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:30.859704    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:31.032639 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.032610    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:31.229494 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.229422    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z"]
Nov 02 23:49:31.230044 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.230027    4294 cpu_manager.go:414] "containerMap: removing container" podUID="1809bac7-6631-4311-a474-758f38821e75" containerName="test-curl-7421-fc6765"
Nov 02 23:49:31.230084 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.230047    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1809bac7-6631-4311-a474-758f38821e75" containerName="test-curl-7421-fc6765"
Nov 02 23:49:31.230212 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.230203    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="1809bac7-6631-4311-a474-758f38821e75" containerName="test-curl-7421-fc6765"
Nov 02 23:49:31.234892 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.234877    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z"
Nov 02 23:49:31.285122 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.285094    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z"]
Nov 02 23:49:31.329184 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.329151    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-3bcb9f\""
Nov 02 23:49:31.329184 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.329181    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-3bcb9f-dockercfg-xbpsh\""
Nov 02 23:49:31.379524 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.379492    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfca5461-e8a7-4538-9e13-26aeca11ed47-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z\" (UID: \"cfca5461-e8a7-4538-9e13-26aeca11ed47\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z"
Nov 02 23:49:31.379703 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.379635    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6pf9\" (UniqueName: \"kubernetes.io/projected/cfca5461-e8a7-4538-9e13-26aeca11ed47-kube-api-access-b6pf9\") pod \"payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z\" (UID: \"cfca5461-e8a7-4538-9e13-26aeca11ed47\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z"
Nov 02 23:49:31.399150 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.399120    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw"]
Nov 02 23:49:31.403431 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.403407    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw"
Nov 02 23:49:31.456696 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.456652    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw"]
Nov 02 23:49:31.481450 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.481379    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4x5x\" (UniqueName: \"kubernetes.io/projected/e1728df9-e8b4-4c64-adc6-9178bdd638d8-kube-api-access-j4x5x\") pod \"payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw\" (UID: \"e1728df9-e8b4-4c64-adc6-9178bdd638d8\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw"
Nov 02 23:49:31.481581 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.481455    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfca5461-e8a7-4538-9e13-26aeca11ed47-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z\" (UID: \"cfca5461-e8a7-4538-9e13-26aeca11ed47\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z"
Nov 02 23:49:31.481658 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.481613    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1728df9-e8b4-4c64-adc6-9178bdd638d8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw\" (UID: \"e1728df9-e8b4-4c64-adc6-9178bdd638d8\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw"
Nov 02 23:49:31.481722 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.481667    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6pf9\" (UniqueName: \"kubernetes.io/projected/cfca5461-e8a7-4538-9e13-26aeca11ed47-kube-api-access-b6pf9\") pod \"payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z\" (UID: \"cfca5461-e8a7-4538-9e13-26aeca11ed47\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z"
Nov 02 23:49:31.482204 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.482184    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfca5461-e8a7-4538-9e13-26aeca11ed47-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z\" (UID: \"cfca5461-e8a7-4538-9e13-26aeca11ed47\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z"
Nov 02 23:49:31.536438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.536412    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6pf9\" (UniqueName: \"kubernetes.io/projected/cfca5461-e8a7-4538-9e13-26aeca11ed47-kube-api-access-b6pf9\") pod \"payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z\" (UID: \"cfca5461-e8a7-4538-9e13-26aeca11ed47\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z"
Nov 02 23:49:31.548484 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.548462    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z"
Nov 02 23:49:31.583108 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.583081    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1728df9-e8b4-4c64-adc6-9178bdd638d8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw\" (UID: \"e1728df9-e8b4-4c64-adc6-9178bdd638d8\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw"
Nov 02 23:49:31.583285 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.583224    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4x5x\" (UniqueName: \"kubernetes.io/projected/e1728df9-e8b4-4c64-adc6-9178bdd638d8-kube-api-access-j4x5x\") pod \"payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw\" (UID: \"e1728df9-e8b4-4c64-adc6-9178bdd638d8\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw"
Nov 02 23:49:31.583948 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.583925    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1728df9-e8b4-4c64-adc6-9178bdd638d8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw\" (UID: \"e1728df9-e8b4-4c64-adc6-9178bdd638d8\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw"
Nov 02 23:49:31.638009 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.637918    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4x5x\" (UniqueName: \"kubernetes.io/projected/e1728df9-e8b4-4c64-adc6-9178bdd638d8-kube-api-access-j4x5x\") pod \"payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw\" (UID: \"e1728df9-e8b4-4c64-adc6-9178bdd638d8\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw"
Nov 02 23:49:31.718290 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.718260    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw"
Nov 02 23:49:31.918919 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:49:31.918889    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfca5461_e8a7_4538_9e13_26aeca11ed47.slice/crio-be1629e64df98f5aad4516f87961632f75837ac96ff617a5bf2ddc42c4c2a94a WatchSource:0}: Error finding container be1629e64df98f5aad4516f87961632f75837ac96ff617a5bf2ddc42c4c2a94a: Status 404 returned error can't find the container with id be1629e64df98f5aad4516f87961632f75837ac96ff617a5bf2ddc42c4c2a94a
Nov 02 23:49:31.919281 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:31.919223    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z"]
Nov 02 23:49:32.034423 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:32.034389    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z" event={"ID":"cfca5461-e8a7-4538-9e13-26aeca11ed47","Type":"ContainerStarted","Data":"4597d2dfc0cf0e2eb801e05cf5138747558b38ebeec98bcc87e547d6285d3148"}
Nov 02 23:49:32.034627 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:32.034470    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z"
Nov 02 23:49:32.034627 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:32.034489    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z" event={"ID":"cfca5461-e8a7-4538-9e13-26aeca11ed47","Type":"ContainerStarted","Data":"be1629e64df98f5aad4516f87961632f75837ac96ff617a5bf2ddc42c4c2a94a"}
Nov 02 23:49:32.091059 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:49:32.091025    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1728df9_e8b4_4c64_adc6_9178bdd638d8.slice/crio-39568e187638f622575c87d17e4e2c0fae6cdea08c08b0fe5e46697c6565b57d WatchSource:0}: Error finding container 39568e187638f622575c87d17e4e2c0fae6cdea08c08b0fe5e46697c6565b57d: Status 404 returned error can't find the container with id 39568e187638f622575c87d17e4e2c0fae6cdea08c08b0fe5e46697c6565b57d
Nov 02 23:49:32.091400 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:32.091373    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw"]
Nov 02 23:49:32.146199 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:32.146155    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z" podStartSLOduration=1.146141271 podStartE2EDuration="1.146141271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:49:32.145408867 +0000 UTC m=+3411.713763668" watchObservedRunningTime="2026-11-02 23:49:32.146141271 +0000 UTC m=+3411.714496059"
Nov 02 23:49:33.043404 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:33.043366    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw" event={"ID":"e1728df9-e8b4-4c64-adc6-9178bdd638d8","Type":"ContainerStarted","Data":"f7915e6d7cdb6b54717f695f9fcd8071ff31168e1ab827f6542cf03ed22dcacf"}
Nov 02 23:49:33.043841 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:33.043432    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw" event={"ID":"e1728df9-e8b4-4c64-adc6-9178bdd638d8","Type":"ContainerStarted","Data":"39568e187638f622575c87d17e4e2c0fae6cdea08c08b0fe5e46697c6565b57d"}
Nov 02 23:49:33.043841 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:33.043649    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw"
Nov 02 23:49:33.152583 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:33.152530    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw" podStartSLOduration=2.152516657 podStartE2EDuration="2.152516657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:49:33.150723469 +0000 UTC m=+3412.719078257" watchObservedRunningTime="2026-11-02 23:49:33.152516657 +0000 UTC m=+3412.720871473"
Nov 02 23:49:34.497512 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:34.497481    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:36.297958 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:36.297884    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:37.768852 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:37.768789    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:38.382474 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:38.382442    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:38.654098 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:38.654018    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:39.070625 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:39.070588    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:40.829764 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:40.829729    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:41.750934 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:49:41.750897    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:49:42.710740 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:42.710707    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:42.887541 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:42.887509    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:43.046556 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:43.046523    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z"
Nov 02 23:49:43.108966 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:43.108932    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:43.860118 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:43.860085    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:44.054862 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:44.054834    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw"
Nov 02 23:49:45.836449 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:45.836418    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:46.006881 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:46.006846    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:46.364655 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:46.364625    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99"]
Nov 02 23:49:46.368488 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:46.368473    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99"
Nov 02 23:49:46.420424 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:46.420397    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99"]
Nov 02 23:49:46.463908 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:46.463877    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-8c2bff\""
Nov 02 23:49:46.464125 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:46.463937    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-8c2bff-dockercfg-x6gf2\""
Nov 02 23:49:46.466628 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:46.466606    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25addf9e-2173-4971-9112-02e9857a65d4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99\" (UID: \"25addf9e-2173-4971-9112-02e9857a65d4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99"
Nov 02 23:49:46.466806 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:46.466790    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tmqs\" (UniqueName: \"kubernetes.io/projected/25addf9e-2173-4971-9112-02e9857a65d4-kube-api-access-6tmqs\") pod \"payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99\" (UID: \"25addf9e-2173-4971-9112-02e9857a65d4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99"
Nov 02 23:49:46.568403 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:46.568370    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25addf9e-2173-4971-9112-02e9857a65d4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99\" (UID: \"25addf9e-2173-4971-9112-02e9857a65d4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99"
Nov 02 23:49:46.568742 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:46.568724    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tmqs\" (UniqueName: \"kubernetes.io/projected/25addf9e-2173-4971-9112-02e9857a65d4-kube-api-access-6tmqs\") pod \"payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99\" (UID: \"25addf9e-2173-4971-9112-02e9857a65d4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99"
Nov 02 23:49:46.569205 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:46.569180    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25addf9e-2173-4971-9112-02e9857a65d4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99\" (UID: \"25addf9e-2173-4971-9112-02e9857a65d4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99"
Nov 02 23:49:46.622974 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:46.622904    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tmqs\" (UniqueName: \"kubernetes.io/projected/25addf9e-2173-4971-9112-02e9857a65d4-kube-api-access-6tmqs\") pod \"payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99\" (UID: \"25addf9e-2173-4971-9112-02e9857a65d4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99"
Nov 02 23:49:46.681754 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:46.681722    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99"
Nov 02 23:49:47.049864 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:47.049831    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99"]
Nov 02 23:49:47.050267 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:49:47.050042    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25addf9e_2173_4971_9112_02e9857a65d4.slice/crio-e69b24c293dceb2d1debfc745315bd8852462ff37e5e910a370192a9c02a7624 WatchSource:0}: Error finding container e69b24c293dceb2d1debfc745315bd8852462ff37e5e910a370192a9c02a7624: Status 404 returned error can't find the container with id e69b24c293dceb2d1debfc745315bd8852462ff37e5e910a370192a9c02a7624
Nov 02 23:49:47.137968 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:47.137939    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99" event={"ID":"25addf9e-2173-4971-9112-02e9857a65d4","Type":"ContainerStarted","Data":"e69b24c293dceb2d1debfc745315bd8852462ff37e5e910a370192a9c02a7624"}
Nov 02 23:49:48.145552 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:48.145517    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99" event={"ID":"25addf9e-2173-4971-9112-02e9857a65d4","Type":"ContainerStarted","Data":"fafd33d687b5dc874eb61bdfc3a52aa6f11982f5de45fcfde8b87876434d66c4"}
Nov 02 23:49:48.146014 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:48.145663    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99"
Nov 02 23:49:48.258118 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:48.258062    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99" podStartSLOduration=2.258048979 podStartE2EDuration="2.258048979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:49:48.255546221 +0000 UTC m=+3427.823901012" watchObservedRunningTime="2026-11-02 23:49:48.258048979 +0000 UTC m=+3427.826403766"
Nov 02 23:49:48.544156 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:48.544123    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:53.041353 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:53.041319    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:57.516577 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:57.516541    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:49:59.154671 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:49:59.154637    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99"
Nov 02 23:50:02.180506 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:02.180473    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:06.687746 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:06.687710    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:11.174603 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:11.174574    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:11.751066 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:50:11.751031    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:50:15.655777 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:15.655744    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:17.332310 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:17.332278    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"]
Nov 02 23:50:17.382286 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:17.382244    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:20.170334 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:20.170288    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:20.357713 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:20.357674    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7418-194d1f"]
Nov 02 23:50:20.364295 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:20.364265    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7418-194d1f"
Nov 02 23:50:20.392033 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:20.391999    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:20.415250 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:20.415215    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7418-194d1f"]
Nov 02 23:50:20.458029 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:20.457941    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wbwlb\""
Nov 02 23:50:20.477466 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:20.477435    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrjx6\" (UniqueName: \"kubernetes.io/projected/07bbdea3-a29a-43a0-a68f-c64683062d81-kube-api-access-xrjx6\") pod \"test-curl-7418-194d1f\" (UID: \"07bbdea3-a29a-43a0-a68f-c64683062d81\") " pod="opendatahub/test-curl-7418-194d1f"
Nov 02 23:50:20.578716 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:20.578688    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrjx6\" (UniqueName: \"kubernetes.io/projected/07bbdea3-a29a-43a0-a68f-c64683062d81-kube-api-access-xrjx6\") pod \"test-curl-7418-194d1f\" (UID: \"07bbdea3-a29a-43a0-a68f-c64683062d81\") " pod="opendatahub/test-curl-7418-194d1f"
Nov 02 23:50:20.633208 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:20.633182    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrjx6\" (UniqueName: \"kubernetes.io/projected/07bbdea3-a29a-43a0-a68f-c64683062d81-kube-api-access-xrjx6\") pod \"test-curl-7418-194d1f\" (UID: \"07bbdea3-a29a-43a0-a68f-c64683062d81\") " pod="opendatahub/test-curl-7418-194d1f"
Nov 02 23:50:20.679246 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:20.679216    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7418-194d1f"
Nov 02 23:50:21.052813 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:21.052786    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7418-194d1f"]
Nov 02 23:50:21.053613 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:50:21.053579    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07bbdea3_a29a_43a0_a68f_c64683062d81.slice/crio-4cda4cba5c11985983c237b800a958d3aa64330393d50e18711c7aabbe94b3f9 WatchSource:0}: Error finding container 4cda4cba5c11985983c237b800a958d3aa64330393d50e18711c7aabbe94b3f9: Status 404 returned error can't find the container with id 4cda4cba5c11985983c237b800a958d3aa64330393d50e18711c7aabbe94b3f9
Nov 02 23:50:21.354058 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:21.353916    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7418-194d1f" event={"ID":"07bbdea3-a29a-43a0-a68f-c64683062d81","Type":"ContainerStarted","Data":"e74baaba6a6324d22192ea073e50c1cc15c1b0d62627ae9958030122f901148f"}
Nov 02 23:50:21.354058 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:21.353966    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7418-194d1f" event={"ID":"07bbdea3-a29a-43a0-a68f-c64683062d81","Type":"ContainerStarted","Data":"4cda4cba5c11985983c237b800a958d3aa64330393d50e18711c7aabbe94b3f9"}
Nov 02 23:50:21.461666 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:21.461615    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7418-194d1f" podStartSLOduration=1.461601193 podStartE2EDuration="1.461601193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:50:21.460553991 +0000 UTC m=+3461.028908777" watchObservedRunningTime="2026-11-02 23:50:21.461601193 +0000 UTC m=+3461.029955979"
Nov 02 23:50:22.822161 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:22.821467    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7418-194d1f"]
Nov 02 23:50:22.825403 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:22.825372    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7418-194d1f"]
Nov 02 23:50:23.367309 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:23.367262    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7418-194d1f" podUID="07bbdea3-a29a-43a0-a68f-c64683062d81" containerName="test-curl-7418-194d1f" containerID="cri-o://e74baaba6a6324d22192ea073e50c1cc15c1b0d62627ae9958030122f901148f" gracePeriod=2
Nov 02 23:50:23.416255 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:50:23.416210    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7418-194d1f\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="07bbdea3-a29a-43a0-a68f-c64683062d81" pod="opendatahub/test-curl-7418-194d1f"
Nov 02 23:50:24.372756 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:24.372713    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:24.950187 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:24.950145    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:25.119580 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:25.119541    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:25.289643 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:25.289604    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:25.684445 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:25.684415    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7418-194d1f"
Nov 02 23:50:25.855759 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:25.855685    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07bbdea3-a29a-43a0-a68f-c64683062d81-kube-api-access-xrjx6\" (UniqueName: \"kubernetes.io/projected/07bbdea3-a29a-43a0-a68f-c64683062d81-kube-api-access-xrjx6\") pod \"07bbdea3-a29a-43a0-a68f-c64683062d81\" (UID: \"07bbdea3-a29a-43a0-a68f-c64683062d81\") "
Nov 02 23:50:25.857804 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:25.857770    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07bbdea3-a29a-43a0-a68f-c64683062d81-kube-api-access-xrjx6" pod "07bbdea3-a29a-43a0-a68f-c64683062d81" (UID: "07bbdea3-a29a-43a0-a68f-c64683062d81"). InnerVolumeSpecName "kube-api-access-xrjx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:50:25.957707 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:25.957598    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrjx6\" (UniqueName: \"kubernetes.io/projected/07bbdea3-a29a-43a0-a68f-c64683062d81-kube-api-access-xrjx6\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:50:26.389164 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:26.389132    4294 generic.go:356] "Generic (PLEG): container finished" podID="07bbdea3-a29a-43a0-a68f-c64683062d81" containerID="e74baaba6a6324d22192ea073e50c1cc15c1b0d62627ae9958030122f901148f" exitCode=137
Nov 02 23:50:26.389360 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:26.389177    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7418-194d1f"
Nov 02 23:50:26.389360 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:26.389205    4294 scope.go:122] "RemoveContainer" containerID="e74baaba6a6324d22192ea073e50c1cc15c1b0d62627ae9958030122f901148f"
Nov 02 23:50:26.399345 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:26.399319    4294 scope.go:122] "RemoveContainer" containerID="e74baaba6a6324d22192ea073e50c1cc15c1b0d62627ae9958030122f901148f"
Nov 02 23:50:26.399666 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:50:26.399646    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e74baaba6a6324d22192ea073e50c1cc15c1b0d62627ae9958030122f901148f\": container with ID starting with e74baaba6a6324d22192ea073e50c1cc15c1b0d62627ae9958030122f901148f not found: ID does not exist" containerID="e74baaba6a6324d22192ea073e50c1cc15c1b0d62627ae9958030122f901148f"
Nov 02 23:50:26.399824 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:26.399675    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e74baaba6a6324d22192ea073e50c1cc15c1b0d62627ae9958030122f901148f"} err="failed to get container status \"e74baaba6a6324d22192ea073e50c1cc15c1b0d62627ae9958030122f901148f\": rpc error: code = NotFound desc = could not find container \"e74baaba6a6324d22192ea073e50c1cc15c1b0d62627ae9958030122f901148f\": container with ID starting with e74baaba6a6324d22192ea073e50c1cc15c1b0d62627ae9958030122f901148f not found: ID does not exist"
Nov 02 23:50:26.446659 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:50:26.446618    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7418-194d1f\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="07bbdea3-a29a-43a0-a68f-c64683062d81" pod="opendatahub/test-curl-7418-194d1f"
Nov 02 23:50:26.549640 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:26.549605    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07bbdea3-a29a-43a0-a68f-c64683062d81" path="/var/lib/kubelet/pods/07bbdea3-a29a-43a0-a68f-c64683062d81/volumes"
Nov 02 23:50:28.756647 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:28.756612    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:29.054829 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:29.054743    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:29.232680 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:29.232648    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:29.466403 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:29.466323    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:29.580206 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:29.580173    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7424-c73f5d"]
Nov 02 23:50:29.580810 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:29.580797    4294 cpu_manager.go:414] "containerMap: removing container" podUID="07bbdea3-a29a-43a0-a68f-c64683062d81" containerName="test-curl-7418-194d1f"
Nov 02 23:50:29.580856 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:29.580812    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07bbdea3-a29a-43a0-a68f-c64683062d81" containerName="test-curl-7418-194d1f"
Nov 02 23:50:29.580928 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:29.580920    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="07bbdea3-a29a-43a0-a68f-c64683062d81" containerName="test-curl-7418-194d1f"
Nov 02 23:50:29.586208 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:29.586189    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-c73f5d"
Nov 02 23:50:29.601413 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:29.601385    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjrjk\" (UniqueName: \"kubernetes.io/projected/e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32-kube-api-access-hjrjk\") pod \"test-curl-7424-c73f5d\" (UID: \"e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32\") " pod="opendatahub/test-curl-7424-c73f5d"
Nov 02 23:50:29.636947 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:29.636914    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7424-c73f5d"]
Nov 02 23:50:29.663744 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:29.663703    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:29.679618 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:29.679589    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wbwlb\""
Nov 02 23:50:29.702854 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:29.702819    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjrjk\" (UniqueName: \"kubernetes.io/projected/e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32-kube-api-access-hjrjk\") pod \"test-curl-7424-c73f5d\" (UID: \"e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32\") " pod="opendatahub/test-curl-7424-c73f5d"
Nov 02 23:50:29.759822 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:29.759790    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjrjk\" (UniqueName: \"kubernetes.io/projected/e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32-kube-api-access-hjrjk\") pod \"test-curl-7424-c73f5d\" (UID: \"e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32\") " pod="opendatahub/test-curl-7424-c73f5d"
Nov 02 23:50:29.900149 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:29.900118    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-c73f5d"
Nov 02 23:50:30.271434 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:30.271410    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7424-c73f5d"]
Nov 02 23:50:30.271748 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:50:30.271723    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode78cb5fa_5edf_41b7_a2b4_6c1cc1ccbc32.slice/crio-39a9b4e9de36a9832acd2622e3b7fbc57a44f3c32a01d81b479353d46fe307a3 WatchSource:0}: Error finding container 39a9b4e9de36a9832acd2622e3b7fbc57a44f3c32a01d81b479353d46fe307a3: Status 404 returned error can't find the container with id 39a9b4e9de36a9832acd2622e3b7fbc57a44f3c32a01d81b479353d46fe307a3
Nov 02 23:50:30.417246 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:30.417214    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7424-c73f5d" event={"ID":"e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32","Type":"ContainerStarted","Data":"a23da37d7a39be6ced1abef9a16c5394cfe02018965fe796eb369e95986f61b5"}
Nov 02 23:50:30.417417 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:30.417258    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7424-c73f5d" event={"ID":"e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32","Type":"ContainerStarted","Data":"39a9b4e9de36a9832acd2622e3b7fbc57a44f3c32a01d81b479353d46fe307a3"}
Nov 02 23:50:30.529281 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:30.529167    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7424-c73f5d" podStartSLOduration=1.529152295 podStartE2EDuration="1.529152295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:50:30.526368775 +0000 UTC m=+3470.094723561" watchObservedRunningTime="2026-11-02 23:50:30.529152295 +0000 UTC m=+3470.097507082"
Nov 02 23:50:31.957425 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:31.957384    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7424-c73f5d"]
Nov 02 23:50:31.961299 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:31.961270    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7424-c73f5d"]
Nov 02 23:50:32.353383 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.353342    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r" podUID="a96bffa8-dbcd-4539-b494-b9ac3cc80dc1" containerName="main" containerID="cri-o://9d8ea634e41310eeab057abe3e532b668834f6a5d27ee5727dfbd689f6bed475" gracePeriod=45
Nov 02 23:50:32.428829 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.428787    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7424-c73f5d" podUID="e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32" containerName="test-curl-7424-c73f5d" containerID="cri-o://a23da37d7a39be6ced1abef9a16c5394cfe02018965fe796eb369e95986f61b5" gracePeriod=2
Nov 02 23:50:32.478224 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:50:32.478178    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7424-c73f5d\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32" pod="opendatahub/test-curl-7424-c73f5d"
Nov 02 23:50:32.627703 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.627671    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:50:32.633926 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.633896    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-tmp-dir\") pod \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") "
Nov 02 23:50:32.634079 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.634032    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-home\" (UniqueName: \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-home\") pod \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") "
Nov 02 23:50:32.634136 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.634104    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-tls-certs\" (UniqueName: \"kubernetes.io/secret/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-tls-certs\") pod \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") "
Nov 02 23:50:32.634215 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.634195    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-tmp-dir" pod "a96bffa8-dbcd-4539-b494-b9ac3cc80dc1" (UID: "a96bffa8-dbcd-4539-b494-b9ac3cc80dc1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:50:32.634276 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.634262    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-kube-api-access-48vz9\" (UniqueName: \"kubernetes.io/projected/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-kube-api-access-48vz9\") pod \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") "
Nov 02 23:50:32.634326 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.634272    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-home" pod "a96bffa8-dbcd-4539-b494-b9ac3cc80dc1" (UID: "a96bffa8-dbcd-4539-b494-b9ac3cc80dc1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:50:32.634369 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.634338    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-dshm\") pod \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") "
Nov 02 23:50:32.634423 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.634405    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-model-cache\") pod \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\" (UID: \"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1\") "
Nov 02 23:50:32.634639 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.634608    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-model-cache" pod "a96bffa8-dbcd-4539-b494-b9ac3cc80dc1" (UID: "a96bffa8-dbcd-4539-b494-b9ac3cc80dc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:50:32.635292 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.635041    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:50:32.635292 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.635065    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:50:32.635292 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.635080    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:50:32.636519 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.636491    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-dshm" pod "a96bffa8-dbcd-4539-b494-b9ac3cc80dc1" (UID: "a96bffa8-dbcd-4539-b494-b9ac3cc80dc1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:50:32.636631 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.636599    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-tls-certs" pod "a96bffa8-dbcd-4539-b494-b9ac3cc80dc1" (UID: "a96bffa8-dbcd-4539-b494-b9ac3cc80dc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:50:32.636684 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.636643    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-kube-api-access-48vz9" pod "a96bffa8-dbcd-4539-b494-b9ac3cc80dc1" (UID: "a96bffa8-dbcd-4539-b494-b9ac3cc80dc1"). InnerVolumeSpecName "kube-api-access-48vz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:50:32.736973 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.736691    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:50:32.736973 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.736729    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48vz9\" (UniqueName: \"kubernetes.io/projected/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-kube-api-access-48vz9\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:50:32.736973 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:32.736744    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:50:33.436813 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:33.436776    4294 generic.go:356] "Generic (PLEG): container finished" podID="a96bffa8-dbcd-4539-b494-b9ac3cc80dc1" containerID="9d8ea634e41310eeab057abe3e532b668834f6a5d27ee5727dfbd689f6bed475" exitCode=0
Nov 02 23:50:33.437326 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:33.436827    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"
Nov 02 23:50:33.437326 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:33.436859    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r" event={"ID":"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1","Type":"ContainerDied","Data":"9d8ea634e41310eeab057abe3e532b668834f6a5d27ee5727dfbd689f6bed475"}
Nov 02 23:50:33.437326 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:33.436915    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r" event={"ID":"a96bffa8-dbcd-4539-b494-b9ac3cc80dc1","Type":"ContainerDied","Data":"971788d260e297488d0da7f98a5b23d7240db61ecc8f2252994f69e2ce450cef"}
Nov 02 23:50:33.437326 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:33.436939    4294 scope.go:122] "RemoveContainer" containerID="9d8ea634e41310eeab057abe3e532b668834f6a5d27ee5727dfbd689f6bed475"
Nov 02 23:50:33.447464 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:33.447447    4294 scope.go:122] "RemoveContainer" containerID="9d8ea634e41310eeab057abe3e532b668834f6a5d27ee5727dfbd689f6bed475"
Nov 02 23:50:33.447750 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:50:33.447731    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d8ea634e41310eeab057abe3e532b668834f6a5d27ee5727dfbd689f6bed475\": container with ID starting with 9d8ea634e41310eeab057abe3e532b668834f6a5d27ee5727dfbd689f6bed475 not found: ID does not exist" containerID="9d8ea634e41310eeab057abe3e532b668834f6a5d27ee5727dfbd689f6bed475"
Nov 02 23:50:33.447789 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:33.447758    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d8ea634e41310eeab057abe3e532b668834f6a5d27ee5727dfbd689f6bed475"} err="failed to get container status \"9d8ea634e41310eeab057abe3e532b668834f6a5d27ee5727dfbd689f6bed475\": rpc error: code = NotFound desc = could not find container \"9d8ea634e41310eeab057abe3e532b668834f6a5d27ee5727dfbd689f6bed475\": container with ID starting with 9d8ea634e41310eeab057abe3e532b668834f6a5d27ee5727dfbd689f6bed475 not found: ID does not exist"
Nov 02 23:50:33.565272 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:33.565223    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r" podUID="a96bffa8-dbcd-4539-b494-b9ac3cc80dc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.147:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:50:33.595338 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:33.595297    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"]
Nov 02 23:50:33.642209 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:33.642174    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fe63ac/auth-test-model-fe63ac-kserve-7f97fcff6f-fk59r"]
Nov 02 23:50:33.778195 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:33.778133    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:33.941690 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:33.941657    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:34.152220 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.152134    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9"]
Nov 02 23:50:34.152768 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.152754    4294 cpu_manager.go:414] "containerMap: removing container" podUID="a96bffa8-dbcd-4539-b494-b9ac3cc80dc1" containerName="main"
Nov 02 23:50:34.152820 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.152770    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a96bffa8-dbcd-4539-b494-b9ac3cc80dc1" containerName="main"
Nov 02 23:50:34.152820 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.152812    4294 cpu_manager.go:414] "containerMap: removing container" podUID="e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32" containerName="test-curl-7424-c73f5d"
Nov 02 23:50:34.152820 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.152817    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32" containerName="test-curl-7424-c73f5d"
Nov 02 23:50:34.152919 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.152904    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32" containerName="test-curl-7424-c73f5d"
Nov 02 23:50:34.152919 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.152915    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a96bffa8-dbcd-4539-b494-b9ac3cc80dc1" containerName="main"
Nov 02 23:50:34.157613 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.157594    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9"
Nov 02 23:50:34.207460 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.207385    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-e65845\""
Nov 02 23:50:34.208241 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.208216    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9"]
Nov 02 23:50:34.217545 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.217511    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-e65845-dockercfg-kw5vq\""
Nov 02 23:50:34.254415 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.254379    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24256826-ee09-4791-af71-55d2e820dfdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9\" (UID: \"24256826-ee09-4791-af71-55d2e820dfdc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9"
Nov 02 23:50:34.254601 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.254429    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kg4r\" (UniqueName: \"kubernetes.io/projected/24256826-ee09-4791-af71-55d2e820dfdc-kube-api-access-4kg4r\") pod \"payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9\" (UID: \"24256826-ee09-4791-af71-55d2e820dfdc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9"
Nov 02 23:50:34.261832 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.261801    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:34.309492 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.309454    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk"]
Nov 02 23:50:34.313851 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.313822    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk"
Nov 02 23:50:34.355893 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.355859    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33dd7ffb-c342-476c-8ab6-6370fff64b9b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk\" (UID: \"33dd7ffb-c342-476c-8ab6-6370fff64b9b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk"
Nov 02 23:50:34.356122 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.356096    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42xzz\" (UniqueName: \"kubernetes.io/projected/33dd7ffb-c342-476c-8ab6-6370fff64b9b-kube-api-access-42xzz\") pod \"payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk\" (UID: \"33dd7ffb-c342-476c-8ab6-6370fff64b9b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk"
Nov 02 23:50:34.356174 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.356139    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24256826-ee09-4791-af71-55d2e820dfdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9\" (UID: \"24256826-ee09-4791-af71-55d2e820dfdc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9"
Nov 02 23:50:34.356210 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.356198    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kg4r\" (UniqueName: \"kubernetes.io/projected/24256826-ee09-4791-af71-55d2e820dfdc-kube-api-access-4kg4r\") pod \"payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9\" (UID: \"24256826-ee09-4791-af71-55d2e820dfdc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9"
Nov 02 23:50:34.356891 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.356873    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24256826-ee09-4791-af71-55d2e820dfdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9\" (UID: \"24256826-ee09-4791-af71-55d2e820dfdc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9"
Nov 02 23:50:34.365166 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.365128    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk"]
Nov 02 23:50:34.410997 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.410892    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kg4r\" (UniqueName: \"kubernetes.io/projected/24256826-ee09-4791-af71-55d2e820dfdc-kube-api-access-4kg4r\") pod \"payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9\" (UID: \"24256826-ee09-4791-af71-55d2e820dfdc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9"
Nov 02 23:50:34.420502 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.420471    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:34.457669 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.457637    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42xzz\" (UniqueName: \"kubernetes.io/projected/33dd7ffb-c342-476c-8ab6-6370fff64b9b-kube-api-access-42xzz\") pod \"payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk\" (UID: \"33dd7ffb-c342-476c-8ab6-6370fff64b9b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk"
Nov 02 23:50:34.458129 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.457725    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33dd7ffb-c342-476c-8ab6-6370fff64b9b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk\" (UID: \"33dd7ffb-c342-476c-8ab6-6370fff64b9b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk"
Nov 02 23:50:34.458459 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.458437    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33dd7ffb-c342-476c-8ab6-6370fff64b9b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk\" (UID: \"33dd7ffb-c342-476c-8ab6-6370fff64b9b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk"
Nov 02 23:50:34.472142 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.472113    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9"
Nov 02 23:50:34.513185 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.513145    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42xzz\" (UniqueName: \"kubernetes.io/projected/33dd7ffb-c342-476c-8ab6-6370fff64b9b-kube-api-access-42xzz\") pod \"payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk\" (UID: \"33dd7ffb-c342-476c-8ab6-6370fff64b9b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk"
Nov 02 23:50:34.553520 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.553483    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a96bffa8-dbcd-4539-b494-b9ac3cc80dc1" path="/var/lib/kubelet/pods/a96bffa8-dbcd-4539-b494-b9ac3cc80dc1/volumes"
Nov 02 23:50:34.585893 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.585864    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:34.630014 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.629972    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk"
Nov 02 23:50:34.674382 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.674093    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-c73f5d"
Nov 02 23:50:34.762125 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.762092    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32-kube-api-access-hjrjk\" (UniqueName: \"kubernetes.io/projected/e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32-kube-api-access-hjrjk\") pod \"e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32\" (UID: \"e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32\") "
Nov 02 23:50:34.764379 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.764354    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32-kube-api-access-hjrjk" pod "e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32" (UID: "e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32"). InnerVolumeSpecName "kube-api-access-hjrjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:50:34.851987 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.851944    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9"]
Nov 02 23:50:34.852174 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:50:34.852149    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24256826_ee09_4791_af71_55d2e820dfdc.slice/crio-aeba83e199b5c7b46ad7fa998305fb61de194b4def4ff5cfa06210cdc342d8c2 WatchSource:0}: Error finding container aeba83e199b5c7b46ad7fa998305fb61de194b4def4ff5cfa06210cdc342d8c2: Status 404 returned error can't find the container with id aeba83e199b5c7b46ad7fa998305fb61de194b4def4ff5cfa06210cdc342d8c2
Nov 02 23:50:34.864134 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:34.864102    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjrjk\" (UniqueName: \"kubernetes.io/projected/e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32-kube-api-access-hjrjk\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:50:35.006071 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:35.006035    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk"]
Nov 02 23:50:35.453053 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:35.453019    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk" event={"ID":"33dd7ffb-c342-476c-8ab6-6370fff64b9b","Type":"ContainerStarted","Data":"64bab326a8c42e05d3c0fd0fdef7294659ca9a18d7247e3e52cb810ae0613282"}
Nov 02 23:50:35.453245 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:35.453079    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk" event={"ID":"33dd7ffb-c342-476c-8ab6-6370fff64b9b","Type":"ContainerStarted","Data":"da6923f1f29b63c1fb7dfe3e642d94ae9889a5df200e43ea320a50f1fab0931d"}
Nov 02 23:50:35.453245 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:35.453105    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk"
Nov 02 23:50:35.454392 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:35.454366    4294 generic.go:356] "Generic (PLEG): container finished" podID="e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32" containerID="a23da37d7a39be6ced1abef9a16c5394cfe02018965fe796eb369e95986f61b5" exitCode=137
Nov 02 23:50:35.454527 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:35.454416    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7424-c73f5d"
Nov 02 23:50:35.454527 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:35.454462    4294 scope.go:122] "RemoveContainer" containerID="a23da37d7a39be6ced1abef9a16c5394cfe02018965fe796eb369e95986f61b5"
Nov 02 23:50:35.456152 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:35.456129    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9" event={"ID":"24256826-ee09-4791-af71-55d2e820dfdc","Type":"ContainerStarted","Data":"0557518c82cc5e9ba825e00ac93eb0c465ba992eab6efe167ee331ff7df1423e"}
Nov 02 23:50:35.456241 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:35.456176    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9" event={"ID":"24256826-ee09-4791-af71-55d2e820dfdc","Type":"ContainerStarted","Data":"aeba83e199b5c7b46ad7fa998305fb61de194b4def4ff5cfa06210cdc342d8c2"}
Nov 02 23:50:35.456241 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:35.456200    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9"
Nov 02 23:50:35.464130 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:35.463964    4294 scope.go:122] "RemoveContainer" containerID="a23da37d7a39be6ced1abef9a16c5394cfe02018965fe796eb369e95986f61b5"
Nov 02 23:50:35.464370 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:50:35.464292    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a23da37d7a39be6ced1abef9a16c5394cfe02018965fe796eb369e95986f61b5\": container with ID starting with a23da37d7a39be6ced1abef9a16c5394cfe02018965fe796eb369e95986f61b5 not found: ID does not exist" containerID="a23da37d7a39be6ced1abef9a16c5394cfe02018965fe796eb369e95986f61b5"
Nov 02 23:50:35.464370 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:35.464318    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23da37d7a39be6ced1abef9a16c5394cfe02018965fe796eb369e95986f61b5"} err="failed to get container status \"a23da37d7a39be6ced1abef9a16c5394cfe02018965fe796eb369e95986f61b5\": rpc error: code = NotFound desc = could not find container \"a23da37d7a39be6ced1abef9a16c5394cfe02018965fe796eb369e95986f61b5\": container with ID starting with a23da37d7a39be6ced1abef9a16c5394cfe02018965fe796eb369e95986f61b5 not found: ID does not exist"
Nov 02 23:50:35.562374 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:35.562319    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk" podStartSLOduration=1.562305624 podStartE2EDuration="1.562305624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:50:35.56104493 +0000 UTC m=+3475.129399717" watchObservedRunningTime="2026-11-02 23:50:35.562305624 +0000 UTC m=+3475.130660429"
Nov 02 23:50:35.609854 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:50:35.609818    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7424-c73f5d\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32" pod="opendatahub/test-curl-7424-c73f5d"
Nov 02 23:50:35.722055 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:35.721921    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9" podStartSLOduration=1.721907756 podStartE2EDuration="1.721907756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:50:35.719739275 +0000 UTC m=+3475.288094100" watchObservedRunningTime="2026-11-02 23:50:35.721907756 +0000 UTC m=+3475.290262542"
Nov 02 23:50:36.548961 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:36.548927    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32" path="/var/lib/kubelet/pods/e78cb5fa-5edf-41b7-a2b4-6c1cc1ccbc32/volumes"
Nov 02 23:50:39.013053 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:39.013016    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:41.142154 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:41.142116    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:41.751764 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:50:41.751725    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:50:42.747509 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:42.747473    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:43.534060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:43.534026    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:46.466282 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:46.466254    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9"
Nov 02 23:50:46.466672 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:46.466320    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk"
Nov 02 23:50:48.064849 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:48.064820    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:52.586439 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:52.586401    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:52.757103 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:52.757069    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:52.952392 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:52.952316    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:50:57.348706 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:50:57.348666    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:00.663393 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:00.663358    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:01.224417 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:01.224387    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:01.410577 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:01.410539    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:02.039056 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:02.039022    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:04.030938 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:04.030899    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"]
Nov 02 23:51:04.072180 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:04.072149    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:04.801539 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:04.801507    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:05.142202 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:05.142111    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:06.845085 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:06.845050    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:08.231536 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:08.231499    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:08.399866 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:08.399832    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:08.607742 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:08.607711    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq"]
Nov 02 23:51:08.612901 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:08.612885    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq"
Nov 02 23:51:08.658992 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:08.658953    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f7c07f-347c-4986-8cb5-ebc8bdb082bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq\" (UID: \"71f7c07f-347c-4986-8cb5-ebc8bdb082bb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq"
Nov 02 23:51:08.659104 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:08.659048    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wppqn\" (UniqueName: \"kubernetes.io/projected/71f7c07f-347c-4986-8cb5-ebc8bdb082bb-kube-api-access-wppqn\") pod \"payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq\" (UID: \"71f7c07f-347c-4986-8cb5-ebc8bdb082bb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq"
Nov 02 23:51:08.662937 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:08.662913    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq"]
Nov 02 23:51:08.706335 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:08.706296    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-f2df33\""
Nov 02 23:51:08.706335 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:08.706318    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-f2df33-dockercfg-nld9j\""
Nov 02 23:51:08.760623 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:08.760594    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f7c07f-347c-4986-8cb5-ebc8bdb082bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq\" (UID: \"71f7c07f-347c-4986-8cb5-ebc8bdb082bb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq"
Nov 02 23:51:08.760810 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:08.760651    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wppqn\" (UniqueName: \"kubernetes.io/projected/71f7c07f-347c-4986-8cb5-ebc8bdb082bb-kube-api-access-wppqn\") pod \"payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq\" (UID: \"71f7c07f-347c-4986-8cb5-ebc8bdb082bb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq"
Nov 02 23:51:08.761379 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:08.761359    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f7c07f-347c-4986-8cb5-ebc8bdb082bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq\" (UID: \"71f7c07f-347c-4986-8cb5-ebc8bdb082bb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq"
Nov 02 23:51:08.815029 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:08.814997    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wppqn\" (UniqueName: \"kubernetes.io/projected/71f7c07f-347c-4986-8cb5-ebc8bdb082bb-kube-api-access-wppqn\") pod \"payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq\" (UID: \"71f7c07f-347c-4986-8cb5-ebc8bdb082bb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq"
Nov 02 23:51:08.926448 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:08.926370    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq"
Nov 02 23:51:09.294802 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:09.294773    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq"]
Nov 02 23:51:09.295480 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:51:09.295453    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f7c07f_347c_4986_8cb5_ebc8bdb082bb.slice/crio-a10dae419989d9de5f6017a2725104696008c4b49d576026acfba761301772d3 WatchSource:0}: Error finding container a10dae419989d9de5f6017a2725104696008c4b49d576026acfba761301772d3: Status 404 returned error can't find the container with id a10dae419989d9de5f6017a2725104696008c4b49d576026acfba761301772d3
Nov 02 23:51:09.675931 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:09.675841    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq" event={"ID":"71f7c07f-347c-4986-8cb5-ebc8bdb082bb","Type":"ContainerStarted","Data":"e3613be192d9a2c044811b702cd17cc07237015dd7a7e2aa81a0d42f263cd717"}
Nov 02 23:51:09.675931 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:09.675900    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq" event={"ID":"71f7c07f-347c-4986-8cb5-ebc8bdb082bb","Type":"ContainerStarted","Data":"a10dae419989d9de5f6017a2725104696008c4b49d576026acfba761301772d3"}
Nov 02 23:51:09.676171 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:09.676015    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq"
Nov 02 23:51:09.786529 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:09.786483    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq" podStartSLOduration=1.786470314 podStartE2EDuration="1.786470314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:51:09.785939859 +0000 UTC m=+3509.354294647" watchObservedRunningTime="2026-11-02 23:51:09.786470314 +0000 UTC m=+3509.354825121"
Nov 02 23:51:11.635265 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:11.635234    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:11.750907 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:51:11.750872    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:51:16.153503 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:16.153471    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:19.051447 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.051372    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f" podUID="906b9443-bdbf-49cc-8abb-d67fd25850c7" containerName="main" containerID="cri-o://66d0ecdb79065b161bc2516d65a775b05d73ff958e21a5e268b526d147aa89e6" gracePeriod=45
Nov 02 23:51:19.080923 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:51:19.080896    4294 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod906b9443_bdbf_49cc_8abb_d67fd25850c7.slice/crio-conmon-66d0ecdb79065b161bc2516d65a775b05d73ff958e21a5e268b526d147aa89e6.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 23:51:19.324241 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.324215    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:51:19.479454 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.479425    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-dshm\") pod \"906b9443-bdbf-49cc-8abb-d67fd25850c7\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") "
Nov 02 23:51:19.479608 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.479486    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-model-cache\") pod \"906b9443-bdbf-49cc-8abb-d67fd25850c7\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") "
Nov 02 23:51:19.479608 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.479556    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/906b9443-bdbf-49cc-8abb-d67fd25850c7-kube-api-access-dn9b2\" (UniqueName: \"kubernetes.io/projected/906b9443-bdbf-49cc-8abb-d67fd25850c7-kube-api-access-dn9b2\") pod \"906b9443-bdbf-49cc-8abb-d67fd25850c7\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") "
Nov 02 23:51:19.479608 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.479585    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/906b9443-bdbf-49cc-8abb-d67fd25850c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/906b9443-bdbf-49cc-8abb-d67fd25850c7-tls-certs\") pod \"906b9443-bdbf-49cc-8abb-d67fd25850c7\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") "
Nov 02 23:51:19.479724 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.479625    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-tmp-dir\") pod \"906b9443-bdbf-49cc-8abb-d67fd25850c7\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") "
Nov 02 23:51:19.479724 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.479675    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-home\" (UniqueName: \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-home\") pod \"906b9443-bdbf-49cc-8abb-d67fd25850c7\" (UID: \"906b9443-bdbf-49cc-8abb-d67fd25850c7\") "
Nov 02 23:51:19.479815 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.479761    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-model-cache" pod "906b9443-bdbf-49cc-8abb-d67fd25850c7" (UID: "906b9443-bdbf-49cc-8abb-d67fd25850c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:51:19.479999 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.479936    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-tmp-dir" pod "906b9443-bdbf-49cc-8abb-d67fd25850c7" (UID: "906b9443-bdbf-49cc-8abb-d67fd25850c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:51:19.480146 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.480009    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-home" pod "906b9443-bdbf-49cc-8abb-d67fd25850c7" (UID: "906b9443-bdbf-49cc-8abb-d67fd25850c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:51:19.480224 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.480187    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:51:19.480224 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.480206    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:51:19.480224 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.480216    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:51:19.481840 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.481815    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/906b9443-bdbf-49cc-8abb-d67fd25850c7-kube-api-access-dn9b2" pod "906b9443-bdbf-49cc-8abb-d67fd25850c7" (UID: "906b9443-bdbf-49cc-8abb-d67fd25850c7"). InnerVolumeSpecName "kube-api-access-dn9b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:51:19.481998 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.481968    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-dshm" pod "906b9443-bdbf-49cc-8abb-d67fd25850c7" (UID: "906b9443-bdbf-49cc-8abb-d67fd25850c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:51:19.482059 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.482013    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/906b9443-bdbf-49cc-8abb-d67fd25850c7-tls-certs" pod "906b9443-bdbf-49cc-8abb-d67fd25850c7" (UID: "906b9443-bdbf-49cc-8abb-d67fd25850c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:51:19.581750 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.581689    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/906b9443-bdbf-49cc-8abb-d67fd25850c7-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:51:19.581750 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.581713    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn9b2\" (UniqueName: \"kubernetes.io/projected/906b9443-bdbf-49cc-8abb-d67fd25850c7-kube-api-access-dn9b2\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:51:19.581750 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.581723    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/906b9443-bdbf-49cc-8abb-d67fd25850c7-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:51:19.738748 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.738709    4294 generic.go:356] "Generic (PLEG): container finished" podID="906b9443-bdbf-49cc-8abb-d67fd25850c7" containerID="66d0ecdb79065b161bc2516d65a775b05d73ff958e21a5e268b526d147aa89e6" exitCode=0
Nov 02 23:51:19.738944 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.738756    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"
Nov 02 23:51:19.738944 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.738786    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f" event={"ID":"906b9443-bdbf-49cc-8abb-d67fd25850c7","Type":"ContainerDied","Data":"66d0ecdb79065b161bc2516d65a775b05d73ff958e21a5e268b526d147aa89e6"}
Nov 02 23:51:19.738944 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.738830    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f" event={"ID":"906b9443-bdbf-49cc-8abb-d67fd25850c7","Type":"ContainerDied","Data":"1e6d3ffdbcca7a294139eec274611979dd9cc706a94a118ab034aafa43e2e1ef"}
Nov 02 23:51:19.738944 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.738845    4294 scope.go:122] "RemoveContainer" containerID="66d0ecdb79065b161bc2516d65a775b05d73ff958e21a5e268b526d147aa89e6"
Nov 02 23:51:19.749878 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.749863    4294 scope.go:122] "RemoveContainer" containerID="66d0ecdb79065b161bc2516d65a775b05d73ff958e21a5e268b526d147aa89e6"
Nov 02 23:51:19.750155 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:51:19.750131    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d0ecdb79065b161bc2516d65a775b05d73ff958e21a5e268b526d147aa89e6\": container with ID starting with 66d0ecdb79065b161bc2516d65a775b05d73ff958e21a5e268b526d147aa89e6 not found: ID does not exist" containerID="66d0ecdb79065b161bc2516d65a775b05d73ff958e21a5e268b526d147aa89e6"
Nov 02 23:51:19.750253 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.750161    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d0ecdb79065b161bc2516d65a775b05d73ff958e21a5e268b526d147aa89e6"} err="failed to get container status \"66d0ecdb79065b161bc2516d65a775b05d73ff958e21a5e268b526d147aa89e6\": rpc error: code = NotFound desc = could not find container \"66d0ecdb79065b161bc2516d65a775b05d73ff958e21a5e268b526d147aa89e6\": container with ID starting with 66d0ecdb79065b161bc2516d65a775b05d73ff958e21a5e268b526d147aa89e6 not found: ID does not exist"
Nov 02 23:51:19.897130 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.897057    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"]
Nov 02 23:51:19.944344 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:19.944317    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a51378/auth-test-model-a51378-kserve-84b565f7cc-4vw4f"]
Nov 02 23:51:20.548523 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:20.548491    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="906b9443-bdbf-49cc-8abb-d67fd25850c7" path="/var/lib/kubelet/pods/906b9443-bdbf-49cc-8abb-d67fd25850c7/volumes"
Nov 02 23:51:20.640377 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:20.640348    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:20.687170 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:20.687139    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq"
Nov 02 23:51:23.781187 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:23.781157    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:23.958833 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:23.958802    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:25.131525 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:25.131484    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:29.682284 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:29.682248    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:29.846715 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:29.846683    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:34.385812 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:34.385779    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"]
Nov 02 23:51:34.389672 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:34.389630    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:34.563131 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:34.563098    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:35.000133 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.000097    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:35.733910 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.733871    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"]
Nov 02 23:51:35.734632 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.734612    4294 cpu_manager.go:414] "containerMap: removing container" podUID="906b9443-bdbf-49cc-8abb-d67fd25850c7" containerName="main"
Nov 02 23:51:35.734632 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.734632    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="906b9443-bdbf-49cc-8abb-d67fd25850c7" containerName="main"
Nov 02 23:51:35.734764 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.734746    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="906b9443-bdbf-49cc-8abb-d67fd25850c7" containerName="main"
Nov 02 23:51:35.739505 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.739487    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.808683 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.808651    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"]
Nov 02 23:51:35.837456 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.837427    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-c11b50-openshift-default-dockercfg-fnwg6\""
Nov 02 23:51:35.849074 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.849045    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:35.870061 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.870028    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-token\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.870188 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.870116    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm4b4\" (UniqueName: \"kubernetes.io/projected/248cacc6-b671-459f-b879-6ce8ddbc72c3-kube-api-access-pm4b4\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.870245 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.870224    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-data\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.870296 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.870261    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-envoy\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.870353 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.870295    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-podinfo\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.870440 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.870420    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-workload-certs\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.870508 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.870488    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-workload-socket\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.870552 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.870532    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-credential-socket\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.870607 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.870592    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/248cacc6-b671-459f-b879-6ce8ddbc72c3-istiod-ca-cert\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.971842 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.971813    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-data\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.972084 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.971851    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-envoy\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.972084 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.971876    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-podinfo\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.972084 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.971913    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-workload-certs\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.972084 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.972015    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-workload-socket\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.972084 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.972058    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-credential-socket\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.972438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.972132    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/248cacc6-b671-459f-b879-6ce8ddbc72c3-istiod-ca-cert\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.972438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.972209    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-token\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.972438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.972256    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm4b4\" (UniqueName: \"kubernetes.io/projected/248cacc6-b671-459f-b879-6ce8ddbc72c3-kube-api-access-pm4b4\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.972438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.972285    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-data\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.972438 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.972337    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-workload-certs\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.972717 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.972455    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-workload-socket\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.972717 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.972479    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-credential-socket\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.972849 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.972830    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/248cacc6-b671-459f-b879-6ce8ddbc72c3-istiod-ca-cert\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.974312 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.974287    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-envoy\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:35.974437 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:35.974414    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-podinfo\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:36.028991 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:36.028947    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-token\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:36.029187 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:36.029160    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm4b4\" (UniqueName: \"kubernetes.io/projected/248cacc6-b671-459f-b879-6ce8ddbc72c3-kube-api-access-pm4b4\") pod \"e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") " pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:36.055272 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:36.055244    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:36.172533 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:36.172496    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:36.469843 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:51:36.469818    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248cacc6_b671_459f_b879_6ce8ddbc72c3.slice/crio-fe7a88b04135c29fe7577ef4d05b0205e4b5666021150e85e351b516ae726269 WatchSource:0}: Error finding container fe7a88b04135c29fe7577ef4d05b0205e4b5666021150e85e351b516ae726269: Status 404 returned error can't find the container with id fe7a88b04135c29fe7577ef4d05b0205e4b5666021150e85e351b516ae726269
Nov 02 23:51:36.470515 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:36.470489    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"]
Nov 02 23:51:36.472433 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:36.472402    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:51:36.472512 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:36.472464    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:51:36.472512 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:36.472493    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:51:36.848812 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:36.848775    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7" event={"ID":"248cacc6-b671-459f-b879-6ce8ddbc72c3","Type":"ContainerStarted","Data":"dffe15488bc2fe12c16140db8f75d0dc19402e1913d9050fc3349891b63b615b"}
Nov 02 23:51:36.849250 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:36.848839    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7" event={"ID":"248cacc6-b671-459f-b879-6ce8ddbc72c3","Type":"ContainerStarted","Data":"fe7a88b04135c29fe7577ef4d05b0205e4b5666021150e85e351b516ae726269"}
Nov 02 23:51:36.962856 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:36.962803    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7" podStartSLOduration=1.962790054 podStartE2EDuration="1.962790054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:51:36.962050659 +0000 UTC m=+3536.530405447" watchObservedRunningTime="2026-11-02 23:51:36.962790054 +0000 UTC m=+3536.531144853"
Nov 02 23:51:37.055821 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:37.055789    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:37.061574 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:37.061552    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:37.855638 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:37.855603    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:37.856573 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:37.856556    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:51:39.050102 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:39.050064    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:40.743731 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:40.743692    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:40.919939 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:40.919905    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:41.271138 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:41.271107    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:41.486183 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:41.486150    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:41.751114 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:51:41.751003    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:51:43.863535 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:43.863505    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:44.019075 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:44.019040    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:44.179117 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:44.179026    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:44.342732 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:44.342692    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:44.401657 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:44.401623    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266"]
Nov 02 23:51:44.406099 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:44.406077    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266"
Nov 02 23:51:44.457850 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:44.457772    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266"]
Nov 02 23:51:44.500740 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:44.500705    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-c11b50\""
Nov 02 23:51:44.500740 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:44.500705    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-c11b50-dockercfg-xv5m4\""
Nov 02 23:51:44.579499 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:44.579456    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s5nv\" (UniqueName: \"kubernetes.io/projected/05cb9af5-5f37-40e2-8a25-51702df1c674-kube-api-access-7s5nv\") pod \"payload-processing-e2e-autores-c11b50-857f488765-qq266\" (UID: \"05cb9af5-5f37-40e2-8a25-51702df1c674\") " pod="openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266"
Nov 02 23:51:44.579499 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:44.579501    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05cb9af5-5f37-40e2-8a25-51702df1c674-plugins-config-volume\") pod \"payload-processing-e2e-autores-c11b50-857f488765-qq266\" (UID: \"05cb9af5-5f37-40e2-8a25-51702df1c674\") " pod="openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266"
Nov 02 23:51:44.680433 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:44.680394    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s5nv\" (UniqueName: \"kubernetes.io/projected/05cb9af5-5f37-40e2-8a25-51702df1c674-kube-api-access-7s5nv\") pod \"payload-processing-e2e-autores-c11b50-857f488765-qq266\" (UID: \"05cb9af5-5f37-40e2-8a25-51702df1c674\") " pod="openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266"
Nov 02 23:51:44.680641 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:44.680444    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05cb9af5-5f37-40e2-8a25-51702df1c674-plugins-config-volume\") pod \"payload-processing-e2e-autores-c11b50-857f488765-qq266\" (UID: \"05cb9af5-5f37-40e2-8a25-51702df1c674\") " pod="openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266"
Nov 02 23:51:44.681231 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:44.681211    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05cb9af5-5f37-40e2-8a25-51702df1c674-plugins-config-volume\") pod \"payload-processing-e2e-autores-c11b50-857f488765-qq266\" (UID: \"05cb9af5-5f37-40e2-8a25-51702df1c674\") " pod="openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266"
Nov 02 23:51:44.735324 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:44.735246    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s5nv\" (UniqueName: \"kubernetes.io/projected/05cb9af5-5f37-40e2-8a25-51702df1c674-kube-api-access-7s5nv\") pod \"payload-processing-e2e-autores-c11b50-857f488765-qq266\" (UID: \"05cb9af5-5f37-40e2-8a25-51702df1c674\") " pod="openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266"
Nov 02 23:51:45.022209 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:45.022178    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266"
Nov 02 23:51:45.392726 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:45.392695    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266"]
Nov 02 23:51:45.392958 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:51:45.392937    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05cb9af5_5f37_40e2_8a25_51702df1c674.slice/crio-efb16839c5c7bcadaba9734a29393ae61307605d2cc592101ab81c291b6f43e6 WatchSource:0}: Error finding container efb16839c5c7bcadaba9734a29393ae61307605d2cc592101ab81c291b6f43e6: Status 404 returned error can't find the container with id efb16839c5c7bcadaba9734a29393ae61307605d2cc592101ab81c291b6f43e6
Nov 02 23:51:45.910960 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:45.910923    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266" event={"ID":"05cb9af5-5f37-40e2-8a25-51702df1c674","Type":"ContainerStarted","Data":"f51705e4a61c2e1e53d855de49d1bea12eda01e38df1f42679c3bf3b3dce6c07"}
Nov 02 23:51:45.911165 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:45.911022    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266" event={"ID":"05cb9af5-5f37-40e2-8a25-51702df1c674","Type":"ContainerStarted","Data":"efb16839c5c7bcadaba9734a29393ae61307605d2cc592101ab81c291b6f43e6"}
Nov 02 23:51:45.911165 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:45.911066    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266"
Nov 02 23:51:46.020071 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:46.020019    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266" podStartSLOduration=2.020004353 podStartE2EDuration="2.020004353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:51:46.01858523 +0000 UTC m=+3545.586940017" watchObservedRunningTime="2026-11-02 23:51:46.020004353 +0000 UTC m=+3545.588359142"
Nov 02 23:51:48.866652 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:48.866620    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:49.406293 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.406249    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl" podUID="9cef90d4-bdb1-4b9b-a925-742b2d29ee0e" containerName="main" containerID="cri-o://916a3d5ee19a8c87208dd9f625b86337e86dd887a57b0d2048793bd6e1ab44ab" gracePeriod=45
Nov 02 23:51:49.581959 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.581920    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl" podUID="9cef90d4-bdb1-4b9b-a925-742b2d29ee0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.154:8000/ready\": dial tcp 10.133.0.154:8000: connect: connection refused"
Nov 02 23:51:49.690906 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.690882    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:51:49.748879 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.748851    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-home\" (UniqueName: \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-home\") pod \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") "
Nov 02 23:51:49.749058 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.748894    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-tmp-dir\") pod \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") "
Nov 02 23:51:49.749058 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.748931    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-kube-api-access-pw58q\" (UniqueName: \"kubernetes.io/projected/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-kube-api-access-pw58q\") pod \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") "
Nov 02 23:51:49.749058 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.749001    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-tls-certs\" (UniqueName: \"kubernetes.io/secret/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-tls-certs\") pod \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") "
Nov 02 23:51:49.749058 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.749036    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-dshm\") pod \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") "
Nov 02 23:51:49.749279 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.749120    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-home" pod "9cef90d4-bdb1-4b9b-a925-742b2d29ee0e" (UID: "9cef90d4-bdb1-4b9b-a925-742b2d29ee0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:51:49.749279 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.749149    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-model-cache\") pod \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\" (UID: \"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e\") "
Nov 02 23:51:49.749279 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.749212    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-tmp-dir" pod "9cef90d4-bdb1-4b9b-a925-742b2d29ee0e" (UID: "9cef90d4-bdb1-4b9b-a925-742b2d29ee0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:51:49.749587 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.749561    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-model-cache" pod "9cef90d4-bdb1-4b9b-a925-742b2d29ee0e" (UID: "9cef90d4-bdb1-4b9b-a925-742b2d29ee0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:51:49.749714 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.749686    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:51:49.749714 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.749706    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:51:49.749822 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.749723    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:51:49.751425 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.751382    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-tls-certs" pod "9cef90d4-bdb1-4b9b-a925-742b2d29ee0e" (UID: "9cef90d4-bdb1-4b9b-a925-742b2d29ee0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:51:49.751529 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.751463    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-dshm" pod "9cef90d4-bdb1-4b9b-a925-742b2d29ee0e" (UID: "9cef90d4-bdb1-4b9b-a925-742b2d29ee0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:51:49.751593 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.751518    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-kube-api-access-pw58q" pod "9cef90d4-bdb1-4b9b-a925-742b2d29ee0e" (UID: "9cef90d4-bdb1-4b9b-a925-742b2d29ee0e"). InnerVolumeSpecName "kube-api-access-pw58q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:51:49.850407 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.850380    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw58q\" (UniqueName: \"kubernetes.io/projected/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-kube-api-access-pw58q\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:51:49.850407 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.850406    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:51:49.850577 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.850417    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:51:49.936023 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.935941    4294 generic.go:356] "Generic (PLEG): container finished" podID="9cef90d4-bdb1-4b9b-a925-742b2d29ee0e" containerID="916a3d5ee19a8c87208dd9f625b86337e86dd887a57b0d2048793bd6e1ab44ab" exitCode=0
Nov 02 23:51:49.936023 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.936006    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"
Nov 02 23:51:49.936423 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.936036    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl" event={"ID":"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e","Type":"ContainerDied","Data":"916a3d5ee19a8c87208dd9f625b86337e86dd887a57b0d2048793bd6e1ab44ab"}
Nov 02 23:51:49.936423 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.936086    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl" event={"ID":"9cef90d4-bdb1-4b9b-a925-742b2d29ee0e","Type":"ContainerDied","Data":"0ee982c6fad08bbf44b54674bda4c3108616a463e85d30ecc779baffc643bc1e"}
Nov 02 23:51:49.936423 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.936109    4294 scope.go:122] "RemoveContainer" containerID="916a3d5ee19a8c87208dd9f625b86337e86dd887a57b0d2048793bd6e1ab44ab"
Nov 02 23:51:49.946395 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.946380    4294 scope.go:122] "RemoveContainer" containerID="916a3d5ee19a8c87208dd9f625b86337e86dd887a57b0d2048793bd6e1ab44ab"
Nov 02 23:51:49.946656 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:51:49.946635    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"916a3d5ee19a8c87208dd9f625b86337e86dd887a57b0d2048793bd6e1ab44ab\": container with ID starting with 916a3d5ee19a8c87208dd9f625b86337e86dd887a57b0d2048793bd6e1ab44ab not found: ID does not exist" containerID="916a3d5ee19a8c87208dd9f625b86337e86dd887a57b0d2048793bd6e1ab44ab"
Nov 02 23:51:49.946747 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:49.946663    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916a3d5ee19a8c87208dd9f625b86337e86dd887a57b0d2048793bd6e1ab44ab"} err="failed to get container status \"916a3d5ee19a8c87208dd9f625b86337e86dd887a57b0d2048793bd6e1ab44ab\": rpc error: code = NotFound desc = could not find container \"916a3d5ee19a8c87208dd9f625b86337e86dd887a57b0d2048793bd6e1ab44ab\": container with ID starting with 916a3d5ee19a8c87208dd9f625b86337e86dd887a57b0d2048793bd6e1ab44ab not found: ID does not exist"
Nov 02 23:51:50.093798 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:50.093763    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"]
Nov 02 23:51:50.097368 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:50.097342    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ca60f/auth-test-model-7ca60f-kserve-cbbbcf5d5-s7wjl"]
Nov 02 23:51:50.548659 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:50.548624    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cef90d4-bdb1-4b9b-a925-742b2d29ee0e" path="/var/lib/kubelet/pods/9cef90d4-bdb1-4b9b-a925-742b2d29ee0e/volumes"
Nov 02 23:51:53.785365 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:53.785266    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:51:56.920416 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:56.920377    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266"
Nov 02 23:51:58.775542 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:51:58.775500    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:00.785866 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:00.785832    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:01.964112 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:01.964075    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:02.437487 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.437459    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"]
Nov 02 23:52:02.438165 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.438145    4294 cpu_manager.go:414] "containerMap: removing container" podUID="9cef90d4-bdb1-4b9b-a925-742b2d29ee0e" containerName="main"
Nov 02 23:52:02.438165 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.438165    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cef90d4-bdb1-4b9b-a925-742b2d29ee0e" containerName="main"
Nov 02 23:52:02.438309 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.438290    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cef90d4-bdb1-4b9b-a925-742b2d29ee0e" containerName="main"
Nov 02 23:52:02.442013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.441972    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.501504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.499631    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6kp4\" (UniqueName: \"kubernetes.io/projected/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-kube-api-access-p6kp4\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.501504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.499778    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-workload-socket\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.501504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.499846    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-envoy\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.501504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.500006    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-data\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.501504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.500149    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-podinfo\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.501504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.500218    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-workload-certs\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.501504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.500316    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-token\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.501504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.500420    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istiod-ca-cert\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.501504 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.500498    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-credential-socket\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.550975 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.550940    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"]
Nov 02 23:52:02.559802 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.559773    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:02.583816 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.583787    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-661a20-openshift-default-dockercfg-x9k5q\""
Nov 02 23:52:02.601443 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.601415    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-workload-socket\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.601587 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.601460    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-envoy\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.601587 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.601550    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-data\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.601717 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.601605    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-podinfo\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.601717 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.601650    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-workload-certs\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.601717 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.601707    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-token\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.601881 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.601767    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istiod-ca-cert\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.601881 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.601808    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-credential-socket\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.601881 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.601831    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-workload-socket\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.601881 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.601866    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6kp4\" (UniqueName: \"kubernetes.io/projected/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-kube-api-access-p6kp4\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.602124 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.601972    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-data\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.602124 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.602051    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-workload-certs\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.602233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.602215    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-credential-socket\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.602566 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.602539    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istiod-ca-cert\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.603795 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.603776    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-envoy\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.604225 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.604203    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-podinfo\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.655727 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.655699    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-token\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.655871 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.655779    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6kp4\" (UniqueName: \"kubernetes.io/projected/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-kube-api-access-p6kp4\") pod \"e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") " pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.756025 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.755999    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:02.824022 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:02.823765    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:03.180665 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:52:03.180628    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b342a5_4d24_4147_921c_21cfa6b1f9ee.slice/crio-1444aea81ab5cf3450289c913b9dbc9f288bb08f97b74b1d8ccabde707f21382 WatchSource:0}: Error finding container 1444aea81ab5cf3450289c913b9dbc9f288bb08f97b74b1d8ccabde707f21382: Status 404 returned error can't find the container with id 1444aea81ab5cf3450289c913b9dbc9f288bb08f97b74b1d8ccabde707f21382
Nov 02 23:52:03.181065 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:03.181038    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"]
Nov 02 23:52:03.183567 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:03.183537    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:52:03.183649 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:03.183603    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:52:03.183649 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:03.183631    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:52:03.637516 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:03.637480    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:04.025114 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:04.025078    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6" event={"ID":"d1b342a5-4d24-4147-921c-21cfa6b1f9ee","Type":"ContainerStarted","Data":"9a64dafb8ff57ae4a39871ad4602482e78f8c4485f7b2848efaa92150d76f0c4"}
Nov 02 23:52:04.025322 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:04.025127    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6" event={"ID":"d1b342a5-4d24-4147-921c-21cfa6b1f9ee","Type":"ContainerStarted","Data":"1444aea81ab5cf3450289c913b9dbc9f288bb08f97b74b1d8ccabde707f21382"}
Nov 02 23:52:04.185553 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:04.185499    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6" podStartSLOduration=2.185485697 podStartE2EDuration="2.185485697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:52:04.185031132 +0000 UTC m=+3563.753385918" watchObservedRunningTime="2026-11-02 23:52:04.185485697 +0000 UTC m=+3563.753840548"
Nov 02 23:52:04.756100 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:04.756075    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:04.761948 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:04.761926    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:05.031714 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:05.031638    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:05.032470 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:05.032451    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:52:06.909506 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:06.909472    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:07.084163 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:07.084114    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:07.258580 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:07.258550    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:08.544030 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:08.543967    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:10.040193 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:10.040158    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:10.199915 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:10.199799    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:10.583082 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:10.583049    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn"]
Nov 02 23:52:10.587573 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:10.587555    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn"
Nov 02 23:52:10.640023 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:10.639970    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn"]
Nov 02 23:52:10.682618 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:10.682583    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-661a20\""
Nov 02 23:52:10.682618 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:10.682613    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-661a20-dockercfg-6wn2v\""
Nov 02 23:52:10.707429 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:10.707395    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9grb\" (UniqueName: \"kubernetes.io/projected/3aaec728-9b6b-4d36-af54-3e138d400a01-kube-api-access-w9grb\") pod \"payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn\" (UID: \"3aaec728-9b6b-4d36-af54-3e138d400a01\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn"
Nov 02 23:52:10.707614 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:10.707442    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aaec728-9b6b-4d36-af54-3e138d400a01-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn\" (UID: \"3aaec728-9b6b-4d36-af54-3e138d400a01\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn"
Nov 02 23:52:10.809163 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:10.809130    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9grb\" (UniqueName: \"kubernetes.io/projected/3aaec728-9b6b-4d36-af54-3e138d400a01-kube-api-access-w9grb\") pod \"payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn\" (UID: \"3aaec728-9b6b-4d36-af54-3e138d400a01\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn"
Nov 02 23:52:10.809361 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:10.809179    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aaec728-9b6b-4d36-af54-3e138d400a01-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn\" (UID: \"3aaec728-9b6b-4d36-af54-3e138d400a01\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn"
Nov 02 23:52:10.809975 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:10.809953    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aaec728-9b6b-4d36-af54-3e138d400a01-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn\" (UID: \"3aaec728-9b6b-4d36-af54-3e138d400a01\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn"
Nov 02 23:52:10.863477 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:10.863402    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9grb\" (UniqueName: \"kubernetes.io/projected/3aaec728-9b6b-4d36-af54-3e138d400a01-kube-api-access-w9grb\") pod \"payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn\" (UID: \"3aaec728-9b6b-4d36-af54-3e138d400a01\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn"
Nov 02 23:52:10.901131 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:10.901097    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn"
Nov 02 23:52:11.270231 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:11.270199    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn"]
Nov 02 23:52:11.271121 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:52:11.271093    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aaec728_9b6b_4d36_af54_3e138d400a01.slice/crio-173d688e76d74ccb816175a0fa3735272d8b8545022e03332d9dd44c62c34f64 WatchSource:0}: Error finding container 173d688e76d74ccb816175a0fa3735272d8b8545022e03332d9dd44c62c34f64: Status 404 returned error can't find the container with id 173d688e76d74ccb816175a0fa3735272d8b8545022e03332d9dd44c62c34f64
Nov 02 23:52:11.752697 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:52:11.752648    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:52:12.076360 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:12.076272    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn" event={"ID":"3aaec728-9b6b-4d36-af54-3e138d400a01","Type":"ContainerStarted","Data":"fa95fa983252d97fa2b39223928d3d877f6f4fea1ac0207777174e317177cceb"}
Nov 02 23:52:12.076360 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:12.076333    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn"
Nov 02 23:52:12.076360 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:12.076356    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn" event={"ID":"3aaec728-9b6b-4d36-af54-3e138d400a01","Type":"ContainerStarted","Data":"173d688e76d74ccb816175a0fa3735272d8b8545022e03332d9dd44c62c34f64"}
Nov 02 23:52:12.185725 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:12.185668    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn" podStartSLOduration=2.18565441 podStartE2EDuration="2.18565441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:52:12.185112592 +0000 UTC m=+3571.753467381" watchObservedRunningTime="2026-11-02 23:52:12.18565441 +0000 UTC m=+3571.754009196"
Nov 02 23:52:13.549950 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:13.549919    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:18.271365 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:18.271317    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:21.107603 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:21.107562    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:22.357062 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:22.357023    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:22.935829 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:22.935792    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:23.085653 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:23.085624    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn"
Nov 02 23:52:23.227454 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:23.227373    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:23.525867 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:23.525828    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:27.036923 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:27.036884    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:27.221864 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:27.221820    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:27.387102 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:27.387008    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:28.008675 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:28.008626    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:28.173787 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:28.173750    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:28.351753 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:28.351668    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:30.982298 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:30.982265    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:31.148662 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:31.148626    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:32.729625 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:32.729592    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:33.160146 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:33.160111    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:37.970896 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:37.970854    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:41.756738 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:52:41.756695    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:52:41.780019 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:41.779972    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:42.478637 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:42.478599    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:46.949507 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:46.949474    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:48.771775 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:48.771741    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:51.571323 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:51.571289    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:52:56.008083 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:52:56.008048    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:00.567949 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:00.567917    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:04.873855 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:04.873822    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"]
Nov 02 23:53:04.878372 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:04.878351    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"
Nov 02 23:53:04.931185 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:04.931148    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"]
Nov 02 23:53:04.974526 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:04.974495    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-c11b50-serving-cert\""
Nov 02 23:53:04.996685 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:04.996655    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z82w5\" (UniqueName: \"kubernetes.io/projected/88859309-dc51-43ee-a461-ec486b3f268f-kube-api-access-z82w5\") pod \"maas-api-e2e-autores-c11b50-8696796f47-5fw9j\" (UID: \"88859309-dc51-43ee-a461-ec486b3f268f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"
Nov 02 23:53:04.996829 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:04.996809    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88859309-dc51-43ee-a461-ec486b3f268f-maas-api-tls\") pod \"maas-api-e2e-autores-c11b50-8696796f47-5fw9j\" (UID: \"88859309-dc51-43ee-a461-ec486b3f268f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"
Nov 02 23:53:04.996886 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:04.996868    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88859309-dc51-43ee-a461-ec486b3f268f-metrics-certs\") pod \"maas-api-e2e-autores-c11b50-8696796f47-5fw9j\" (UID: \"88859309-dc51-43ee-a461-ec486b3f268f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"
Nov 02 23:53:04.997059 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:04.997036    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88859309-dc51-43ee-a461-ec486b3f268f-openshift-service-ca\") pod \"maas-api-e2e-autores-c11b50-8696796f47-5fw9j\" (UID: \"88859309-dc51-43ee-a461-ec486b3f268f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"
Nov 02 23:53:05.098470 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:05.098437    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88859309-dc51-43ee-a461-ec486b3f268f-maas-api-tls\") pod \"maas-api-e2e-autores-c11b50-8696796f47-5fw9j\" (UID: \"88859309-dc51-43ee-a461-ec486b3f268f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"
Nov 02 23:53:05.098669 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:05.098499    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88859309-dc51-43ee-a461-ec486b3f268f-metrics-certs\") pod \"maas-api-e2e-autores-c11b50-8696796f47-5fw9j\" (UID: \"88859309-dc51-43ee-a461-ec486b3f268f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"
Nov 02 23:53:05.098669 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:05.098562    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88859309-dc51-43ee-a461-ec486b3f268f-openshift-service-ca\") pod \"maas-api-e2e-autores-c11b50-8696796f47-5fw9j\" (UID: \"88859309-dc51-43ee-a461-ec486b3f268f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"
Nov 02 23:53:05.098669 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:05.098608    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z82w5\" (UniqueName: \"kubernetes.io/projected/88859309-dc51-43ee-a461-ec486b3f268f-kube-api-access-z82w5\") pod \"maas-api-e2e-autores-c11b50-8696796f47-5fw9j\" (UID: \"88859309-dc51-43ee-a461-ec486b3f268f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"
Nov 02 23:53:05.098669 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:53:05.098607    4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-serving-cert: secret "maas-api-e2e-autores-c11b50-serving-cert" not found
Nov 02 23:53:05.098896 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:53:05.098694    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88859309-dc51-43ee-a461-ec486b3f268f-maas-api-tls podName:88859309-dc51-43ee-a461-ec486b3f268f nodeName:}" failed. No retries permitted until 2026-11-02 23:53:05.598673054 +0000 UTC m=+3625.167027833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/88859309-dc51-43ee-a461-ec486b3f268f-maas-api-tls") pod "maas-api-e2e-autores-c11b50-8696796f47-5fw9j" (UID: "88859309-dc51-43ee-a461-ec486b3f268f") : secret "maas-api-e2e-autores-c11b50-serving-cert" not found
Nov 02 23:53:05.099312 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:05.099294    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88859309-dc51-43ee-a461-ec486b3f268f-openshift-service-ca\") pod \"maas-api-e2e-autores-c11b50-8696796f47-5fw9j\" (UID: \"88859309-dc51-43ee-a461-ec486b3f268f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"
Nov 02 23:53:05.100913 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:05.100891    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88859309-dc51-43ee-a461-ec486b3f268f-metrics-certs\") pod \"maas-api-e2e-autores-c11b50-8696796f47-5fw9j\" (UID: \"88859309-dc51-43ee-a461-ec486b3f268f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"
Nov 02 23:53:05.150907 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:05.150811    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:05.153461 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:05.153429    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z82w5\" (UniqueName: \"kubernetes.io/projected/88859309-dc51-43ee-a461-ec486b3f268f-kube-api-access-z82w5\") pod \"maas-api-e2e-autores-c11b50-8696796f47-5fw9j\" (UID: \"88859309-dc51-43ee-a461-ec486b3f268f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"
Nov 02 23:53:05.308355 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:05.308322    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:05.459958 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:05.459873    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:05.605641 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:05.605601    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88859309-dc51-43ee-a461-ec486b3f268f-maas-api-tls\") pod \"maas-api-e2e-autores-c11b50-8696796f47-5fw9j\" (UID: \"88859309-dc51-43ee-a461-ec486b3f268f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"
Nov 02 23:53:05.608043 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:05.608023    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88859309-dc51-43ee-a461-ec486b3f268f-maas-api-tls\") pod \"maas-api-e2e-autores-c11b50-8696796f47-5fw9j\" (UID: \"88859309-dc51-43ee-a461-ec486b3f268f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"
Nov 02 23:53:05.792692 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:05.792656    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"
Nov 02 23:53:06.167563 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:06.167535    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"]
Nov 02 23:53:06.168257 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:53:06.168227    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88859309_dc51_43ee_a461_ec486b3f268f.slice/crio-4d9057e47042737454154b3542c6c17d2238c6e2585720ff84aeb47aa57d9337 WatchSource:0}: Error finding container 4d9057e47042737454154b3542c6c17d2238c6e2585720ff84aeb47aa57d9337: Status 404 returned error can't find the container with id 4d9057e47042737454154b3542c6c17d2238c6e2585720ff84aeb47aa57d9337
Nov 02 23:53:06.448773 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:06.448692    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j" event={"ID":"88859309-dc51-43ee-a461-ec486b3f268f","Type":"ContainerStarted","Data":"4d9057e47042737454154b3542c6c17d2238c6e2585720ff84aeb47aa57d9337"}
Nov 02 23:53:07.457219 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:07.457125    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j" event={"ID":"88859309-dc51-43ee-a461-ec486b3f268f","Type":"ContainerStarted","Data":"a2c03b14127f1cf7647e5b78e6b2092e93378b80f0643d0ac035c005aefd12b3"}
Nov 02 23:53:07.457219 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:07.457216    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"
Nov 02 23:53:07.571872 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:07.571811    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j" podStartSLOduration=2.620621764 podStartE2EDuration="3.571796938s" totalImagesPullingTime="951.175174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:53:07.568381726 +0000 UTC m=+3627.136736514" watchObservedRunningTime="2026-11-02 23:53:07.571796938 +0000 UTC m=+3627.140151724"
Nov 02 23:53:09.783411 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:09.783375    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:11.754786 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:53:11.754752    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:53:13.469372 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:13.469339    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"
Nov 02 23:53:14.309332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:14.309298    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:18.894759 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:18.894726    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:22.920226 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:22.920184    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:23.907532 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:23.907447    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:24.209367 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:24.209280    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:24.802303 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:24.802248    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:25.003892 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:25.003852    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:28.554855 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:28.554788    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:28.746386 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:28.746338    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:31.649639 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:31.649607    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:31.821865 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:31.821828    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:33.462446 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:33.462412    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:38.093358 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.093316    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"]
Nov 02 23:53:38.097781 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.097762    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"
Nov 02 23:53:38.151755 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.151720    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"]
Nov 02 23:53:38.194719 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.194689    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-661a20-serving-cert\""
Nov 02 23:53:38.242486 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.242447    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d129273b-8713-4065-b518-7001a19aefd1-metrics-certs\") pod \"maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p\" (UID: \"d129273b-8713-4065-b518-7001a19aefd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"
Nov 02 23:53:38.242691 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.242539    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d129273b-8713-4065-b518-7001a19aefd1-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p\" (UID: \"d129273b-8713-4065-b518-7001a19aefd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"
Nov 02 23:53:38.242691 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.242647    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d129273b-8713-4065-b518-7001a19aefd1-maas-api-tls\") pod \"maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p\" (UID: \"d129273b-8713-4065-b518-7001a19aefd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"
Nov 02 23:53:38.242819 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.242736    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg4hd\" (UniqueName: \"kubernetes.io/projected/d129273b-8713-4065-b518-7001a19aefd1-kube-api-access-vg4hd\") pod \"maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p\" (UID: \"d129273b-8713-4065-b518-7001a19aefd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"
Nov 02 23:53:38.312800 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.312764    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:38.344030 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.343906    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d129273b-8713-4065-b518-7001a19aefd1-metrics-certs\") pod \"maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p\" (UID: \"d129273b-8713-4065-b518-7001a19aefd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"
Nov 02 23:53:38.344030 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.344006    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d129273b-8713-4065-b518-7001a19aefd1-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p\" (UID: \"d129273b-8713-4065-b518-7001a19aefd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"
Nov 02 23:53:38.344272 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.344048    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d129273b-8713-4065-b518-7001a19aefd1-maas-api-tls\") pod \"maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p\" (UID: \"d129273b-8713-4065-b518-7001a19aefd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"
Nov 02 23:53:38.344272 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.344116    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg4hd\" (UniqueName: \"kubernetes.io/projected/d129273b-8713-4065-b518-7001a19aefd1-kube-api-access-vg4hd\") pod \"maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p\" (UID: \"d129273b-8713-4065-b518-7001a19aefd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"
Nov 02 23:53:38.344801 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.344777    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d129273b-8713-4065-b518-7001a19aefd1-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p\" (UID: \"d129273b-8713-4065-b518-7001a19aefd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"
Nov 02 23:53:38.346749 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.346721    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d129273b-8713-4065-b518-7001a19aefd1-metrics-certs\") pod \"maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p\" (UID: \"d129273b-8713-4065-b518-7001a19aefd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"
Nov 02 23:53:38.346850 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.346808    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d129273b-8713-4065-b518-7001a19aefd1-maas-api-tls\") pod \"maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p\" (UID: \"d129273b-8713-4065-b518-7001a19aefd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"
Nov 02 23:53:38.398605 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.398571    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg4hd\" (UniqueName: \"kubernetes.io/projected/d129273b-8713-4065-b518-7001a19aefd1-kube-api-access-vg4hd\") pod \"maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p\" (UID: \"d129273b-8713-4065-b518-7001a19aefd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"
Nov 02 23:53:38.414067 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.414035    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"
Nov 02 23:53:38.476126 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.474664    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:38.640124 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.640038    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:38.801538 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:53:38.801491    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd129273b_8713_4065_b518_7001a19aefd1.slice/crio-464cf5338ec61aca95e7435c7afe4dd66d32dd5b042947b14eacec8e59b3d717 WatchSource:0}: Error finding container 464cf5338ec61aca95e7435c7afe4dd66d32dd5b042947b14eacec8e59b3d717: Status 404 returned error can't find the container with id 464cf5338ec61aca95e7435c7afe4dd66d32dd5b042947b14eacec8e59b3d717
Nov 02 23:53:38.803149 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.803116    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"]
Nov 02 23:53:38.897218 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.897126    4294 patch_prober.go:29] interesting pod/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 23:53:38.897218 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:38.897194    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7" podUID="248cacc6-b671-459f-b879-6ce8ddbc72c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 23:53:39.668653 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:39.668619    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p" event={"ID":"d129273b-8713-4065-b518-7001a19aefd1","Type":"ContainerStarted","Data":"464cf5338ec61aca95e7435c7afe4dd66d32dd5b042947b14eacec8e59b3d717"}
Nov 02 23:53:40.676553 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:40.676520    4294 generic.go:356] "Generic (PLEG): container finished" podID="248cacc6-b671-459f-b879-6ce8ddbc72c3" containerID="dffe15488bc2fe12c16140db8f75d0dc19402e1913d9050fc3349891b63b615b" exitCode=0
Nov 02 23:53:40.677081 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:40.676590    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7" event={"ID":"248cacc6-b671-459f-b879-6ce8ddbc72c3","Type":"ContainerDied","Data":"dffe15488bc2fe12c16140db8f75d0dc19402e1913d9050fc3349891b63b615b"}
Nov 02 23:53:40.677081 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:40.677014    4294 scope.go:122] "RemoveContainer" containerID="dffe15488bc2fe12c16140db8f75d0dc19402e1913d9050fc3349891b63b615b"
Nov 02 23:53:40.678442 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:40.678421    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p" event={"ID":"d129273b-8713-4065-b518-7001a19aefd1","Type":"ContainerStarted","Data":"4fd25e91763332fb65da0a08abcfa36a4f70e4f333690f706d870d3fea857259"}
Nov 02 23:53:40.678564 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:40.678463    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"
Nov 02 23:53:40.678791 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:40.678763    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:53:40.678865 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:40.678847    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:53:40.678926 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:40.678900    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:53:40.947953 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:40.947844    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p" podStartSLOduration=2.089147337 podStartE2EDuration="2.947824929s" totalImagesPullingTime="858.677592ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:53:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:53:40.947346577 +0000 UTC m=+3660.515701377" watchObservedRunningTime="2026-11-02 23:53:40.947824929 +0000 UTC m=+3660.516179718"
Nov 02 23:53:41.691466 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:41.691427    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7" event={"ID":"248cacc6-b671-459f-b879-6ce8ddbc72c3","Type":"ContainerStarted","Data":"a38dbaf41aa9fcd103b3353bfa27b5d4a359acde4dad86a68296edf7ebe803d9"}
Nov 02 23:53:41.750552 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:53:41.750513    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:53:42.065021 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:42.060143    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:53:42.065267 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:42.065121    4294 patch_prober.go:29] interesting pod/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 02 23:53:42.065267 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:42.065186    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7" podUID="248cacc6-b671-459f-b879-6ce8ddbc72c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 02 23:53:42.178253 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:42.178212    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:43.031851 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:43.031811    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:43.065003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:43.057286    4294 patch_prober.go:29] interesting pod/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 02 23:53:43.065003 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:43.057349    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7" podUID="248cacc6-b671-459f-b879-6ce8ddbc72c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 02 23:53:44.056340 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:44.056299    4294 patch_prober.go:29] interesting pod/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 02 23:53:44.056999 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:44.056948    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7" podUID="248cacc6-b671-459f-b879-6ce8ddbc72c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 02 23:53:45.069005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:45.060374    4294 patch_prober.go:29] interesting pod/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 02 23:53:45.069005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:45.060439    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7" podUID="248cacc6-b671-459f-b879-6ce8ddbc72c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 02 23:53:46.074907 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:46.074868    4294 patch_prober.go:29] interesting pod/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 02 23:53:46.075547 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:46.075511    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7" podUID="248cacc6-b671-459f-b879-6ce8ddbc72c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 02 23:53:46.697705 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:46.697672    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"
Nov 02 23:53:47.059731 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:47.059703    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:53:47.060201 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:47.060185    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:53:47.060748 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:47.060730    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:53:47.752271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:47.752238    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:52.263825 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:52.263794    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:53:56.981868 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:53:56.981825    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:01.379434 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:01.379396    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:01.551234 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:01.551194    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:01.719814 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:01.719726    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:02.847355 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:02.847319    4294 generic.go:356] "Generic (PLEG): container finished" podID="d1b342a5-4d24-4147-921c-21cfa6b1f9ee" containerID="9a64dafb8ff57ae4a39871ad4602482e78f8c4485f7b2848efaa92150d76f0c4" exitCode=0
Nov 02 23:54:02.847832 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:02.847389    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6" event={"ID":"d1b342a5-4d24-4147-921c-21cfa6b1f9ee","Type":"ContainerDied","Data":"9a64dafb8ff57ae4a39871ad4602482e78f8c4485f7b2848efaa92150d76f0c4"}
Nov 02 23:54:02.847832 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:02.847828    4294 scope.go:122] "RemoveContainer" containerID="9a64dafb8ff57ae4a39871ad4602482e78f8c4485f7b2848efaa92150d76f0c4"
Nov 02 23:54:02.849694 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:02.849664    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:54:02.849810 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:02.849738    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:54:02.849810 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:02.849771    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:54:03.880707 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:03.880658    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6" event={"ID":"d1b342a5-4d24-4147-921c-21cfa6b1f9ee","Type":"ContainerStarted","Data":"5c1eec836cd021ad0fd9c5344318055060cdaa3c4db9b1f4032cdf3dea3de055"}
Nov 02 23:54:04.772080 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:04.772043    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:54:04.778309 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:04.778272    4294 patch_prober.go:29] interesting pod/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 23:54:04.778598 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:04.778571    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6" podUID="d1b342a5-4d24-4147-921c-21cfa6b1f9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 23:54:05.032657 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:05.032567    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:54:05.770473 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:05.770430    4294 patch_prober.go:29] interesting pod/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 23:54:05.772197 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:05.772161    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6" podUID="d1b342a5-4d24-4147-921c-21cfa6b1f9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 23:54:06.124076 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:06.123961    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:06.758377 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:06.758332    4294 patch_prober.go:29] interesting pod/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 23:54:06.758719 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:06.758688    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6" podUID="d1b342a5-4d24-4147-921c-21cfa6b1f9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 23:54:07.778480 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:07.778428    4294 patch_prober.go:29] interesting pod/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 23:54:07.779079 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:07.778514    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6" podUID="d1b342a5-4d24-4147-921c-21cfa6b1f9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 23:54:08.598234 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:08.598194    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:08.771005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:08.768409    4294 patch_prober.go:29] interesting pod/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 02 23:54:08.771005 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:08.768501    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6" podUID="d1b342a5-4d24-4147-921c-21cfa6b1f9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 02 23:54:09.762708 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:09.762670    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:54:09.764079 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:09.764054    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:54:10.841543 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:10.841503    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:11.753137 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:54:11.753087    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:54:14.484865 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:14.484831    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:15.446202 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:15.446156    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:16.759765 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:16.759727    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq"]
Nov 02 23:54:17.036226 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:17.036130    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:17.203935 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:17.203884    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:19.937117 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:19.937073    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:20.384598 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:20.384563    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:21.760883 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:21.760833    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq" podUID="71f7c07f-347c-4986-8cb5-ebc8bdb082bb" containerName="payload-pre-processing" containerID="cri-o://e3613be192d9a2c044811b702cd17cc07237015dd7a7e2aa81a0d42f263cd717" gracePeriod=25
Nov 02 23:54:22.030920 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:22.030825    4294 generic.go:356] "Generic (PLEG): container finished" podID="71f7c07f-347c-4986-8cb5-ebc8bdb082bb" containerID="e3613be192d9a2c044811b702cd17cc07237015dd7a7e2aa81a0d42f263cd717" exitCode=0
Nov 02 23:54:22.031108 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:22.030912    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq" event={"ID":"71f7c07f-347c-4986-8cb5-ebc8bdb082bb","Type":"ContainerDied","Data":"e3613be192d9a2c044811b702cd17cc07237015dd7a7e2aa81a0d42f263cd717"}
Nov 02 23:54:22.031108 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:22.031046    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq" event={"ID":"71f7c07f-347c-4986-8cb5-ebc8bdb082bb","Type":"ContainerDied","Data":"a10dae419989d9de5f6017a2725104696008c4b49d576026acfba761301772d3"}
Nov 02 23:54:22.031108 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:22.031072    4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a10dae419989d9de5f6017a2725104696008c4b49d576026acfba761301772d3"
Nov 02 23:54:22.032289 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:22.032272    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq"
Nov 02 23:54:22.143449 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:22.143414    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71f7c07f-347c-4986-8cb5-ebc8bdb082bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f7c07f-347c-4986-8cb5-ebc8bdb082bb-plugins-config-volume\") pod \"71f7c07f-347c-4986-8cb5-ebc8bdb082bb\" (UID: \"71f7c07f-347c-4986-8cb5-ebc8bdb082bb\") "
Nov 02 23:54:22.143720 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:22.143694    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f7c07f-347c-4986-8cb5-ebc8bdb082bb-kube-api-access-wppqn\" (UniqueName: \"kubernetes.io/projected/71f7c07f-347c-4986-8cb5-ebc8bdb082bb-kube-api-access-wppqn\") pod \"71f7c07f-347c-4986-8cb5-ebc8bdb082bb\" (UID: \"71f7c07f-347c-4986-8cb5-ebc8bdb082bb\") "
Nov 02 23:54:22.144495 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:22.144461    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f7c07f-347c-4986-8cb5-ebc8bdb082bb-plugins-config-volume" pod "71f7c07f-347c-4986-8cb5-ebc8bdb082bb" (UID: "71f7c07f-347c-4986-8cb5-ebc8bdb082bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:54:22.147446 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:22.147409    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f7c07f-347c-4986-8cb5-ebc8bdb082bb-kube-api-access-wppqn" pod "71f7c07f-347c-4986-8cb5-ebc8bdb082bb" (UID: "71f7c07f-347c-4986-8cb5-ebc8bdb082bb"). InnerVolumeSpecName "kube-api-access-wppqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:54:22.245878 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:22.245836    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wppqn\" (UniqueName: \"kubernetes.io/projected/71f7c07f-347c-4986-8cb5-ebc8bdb082bb-kube-api-access-wppqn\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:54:22.245878 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:22.245873    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f7c07f-347c-4986-8cb5-ebc8bdb082bb-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:54:23.037279 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:23.037248    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq"
Nov 02 23:54:23.194799 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:23.194764    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq"]
Nov 02 23:54:23.242262 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:23.242227    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f2df33-7585c57455-nxgmq"]
Nov 02 23:54:24.455332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:24.455297    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:24.549423 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:24.549382    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71f7c07f-347c-4986-8cb5-ebc8bdb082bb" path="/var/lib/kubelet/pods/71f7c07f-347c-4986-8cb5-ebc8bdb082bb/volumes"
Nov 02 23:54:29.063578 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:29.063532    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:33.539791 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:33.539755    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:38.031253 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:38.031217    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:41.751496 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:54:41.751450    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:54:42.489505 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:42.489472    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:42.647651 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:42.647618    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:43.771305 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:43.771271    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:47.033850 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:47.033814    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:49.330746 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:49.330710    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266"]
Nov 02 23:54:49.745390 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:49.745357    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:49.896624 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:49.896587    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:50.058537 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:50.058432    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:51.643502 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:51.643468    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:54.331936 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:54.331845    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266" podUID="05cb9af5-5f37-40e2-8a25-51702df1c674" containerName="payload-processing" containerID="cri-o://f51705e4a61c2e1e53d855de49d1bea12eda01e38df1f42679c3bf3b3dce6c07" gracePeriod=25
Nov 02 23:54:54.597799 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:54.597732    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266"
Nov 02 23:54:54.652915 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:54.652883    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05cb9af5-5f37-40e2-8a25-51702df1c674-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05cb9af5-5f37-40e2-8a25-51702df1c674-plugins-config-volume\") pod \"05cb9af5-5f37-40e2-8a25-51702df1c674\" (UID: \"05cb9af5-5f37-40e2-8a25-51702df1c674\") "
Nov 02 23:54:54.653114 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:54.652965    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05cb9af5-5f37-40e2-8a25-51702df1c674-kube-api-access-7s5nv\" (UniqueName: \"kubernetes.io/projected/05cb9af5-5f37-40e2-8a25-51702df1c674-kube-api-access-7s5nv\") pod \"05cb9af5-5f37-40e2-8a25-51702df1c674\" (UID: \"05cb9af5-5f37-40e2-8a25-51702df1c674\") "
Nov 02 23:54:54.653331 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:54.653309    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05cb9af5-5f37-40e2-8a25-51702df1c674-plugins-config-volume" pod "05cb9af5-5f37-40e2-8a25-51702df1c674" (UID: "05cb9af5-5f37-40e2-8a25-51702df1c674"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:54:54.653691 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:54.653666    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05cb9af5-5f37-40e2-8a25-51702df1c674-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:54:54.655588 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:54.655562    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05cb9af5-5f37-40e2-8a25-51702df1c674-kube-api-access-7s5nv" pod "05cb9af5-5f37-40e2-8a25-51702df1c674" (UID: "05cb9af5-5f37-40e2-8a25-51702df1c674"). InnerVolumeSpecName "kube-api-access-7s5nv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:54:54.754565 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:54.754534    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s5nv\" (UniqueName: \"kubernetes.io/projected/05cb9af5-5f37-40e2-8a25-51702df1c674-kube-api-access-7s5nv\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:54:55.250914 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:55.250874    4294 generic.go:356] "Generic (PLEG): container finished" podID="05cb9af5-5f37-40e2-8a25-51702df1c674" containerID="f51705e4a61c2e1e53d855de49d1bea12eda01e38df1f42679c3bf3b3dce6c07" exitCode=0
Nov 02 23:54:55.251147 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:55.250927    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266"
Nov 02 23:54:55.251147 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:55.250959    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266" event={"ID":"05cb9af5-5f37-40e2-8a25-51702df1c674","Type":"ContainerDied","Data":"f51705e4a61c2e1e53d855de49d1bea12eda01e38df1f42679c3bf3b3dce6c07"}
Nov 02 23:54:55.251147 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:55.251027    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266" event={"ID":"05cb9af5-5f37-40e2-8a25-51702df1c674","Type":"ContainerDied","Data":"efb16839c5c7bcadaba9734a29393ae61307605d2cc592101ab81c291b6f43e6"}
Nov 02 23:54:55.251147 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:55.251050    4294 scope.go:122] "RemoveContainer" containerID="f51705e4a61c2e1e53d855de49d1bea12eda01e38df1f42679c3bf3b3dce6c07"
Nov 02 23:54:55.262266 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:55.262250    4294 scope.go:122] "RemoveContainer" containerID="f51705e4a61c2e1e53d855de49d1bea12eda01e38df1f42679c3bf3b3dce6c07"
Nov 02 23:54:55.262532 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:54:55.262513    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f51705e4a61c2e1e53d855de49d1bea12eda01e38df1f42679c3bf3b3dce6c07\": container with ID starting with f51705e4a61c2e1e53d855de49d1bea12eda01e38df1f42679c3bf3b3dce6c07 not found: ID does not exist" containerID="f51705e4a61c2e1e53d855de49d1bea12eda01e38df1f42679c3bf3b3dce6c07"
Nov 02 23:54:55.262600 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:55.262544    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f51705e4a61c2e1e53d855de49d1bea12eda01e38df1f42679c3bf3b3dce6c07"} err="failed to get container status \"f51705e4a61c2e1e53d855de49d1bea12eda01e38df1f42679c3bf3b3dce6c07\": rpc error: code = NotFound desc = could not find container \"f51705e4a61c2e1e53d855de49d1bea12eda01e38df1f42679c3bf3b3dce6c07\": container with ID starting with f51705e4a61c2e1e53d855de49d1bea12eda01e38df1f42679c3bf3b3dce6c07 not found: ID does not exist"
Nov 02 23:54:55.408584 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:55.408548    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266"]
Nov 02 23:54:55.455763 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:55.455727    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c11b50-857f488765-qq266"]
Nov 02 23:54:55.564216 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:55.564136    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:56.205653 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:56.205620    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:54:56.556731 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:54:56.556685    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05cb9af5-5f37-40e2-8a25-51702df1c674" path="/var/lib/kubelet/pods/05cb9af5-5f37-40e2-8a25-51702df1c674/volumes"
Nov 02 23:55:00.704339 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:00.704292    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:04.891724 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:04.891694    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"]
Nov 02 23:55:04.892376 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:04.892356    4294 cpu_manager.go:414] "containerMap: removing container" podUID="05cb9af5-5f37-40e2-8a25-51702df1c674" containerName="payload-processing"
Nov 02 23:55:04.892376 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:04.892372    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05cb9af5-5f37-40e2-8a25-51702df1c674" containerName="payload-processing"
Nov 02 23:55:04.892542 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:04.892405    4294 cpu_manager.go:414] "containerMap: removing container" podUID="71f7c07f-347c-4986-8cb5-ebc8bdb082bb" containerName="payload-pre-processing"
Nov 02 23:55:04.892542 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:04.892410    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f7c07f-347c-4986-8cb5-ebc8bdb082bb" containerName="payload-pre-processing"
Nov 02 23:55:04.892542 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:04.892503    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f7c07f-347c-4986-8cb5-ebc8bdb082bb" containerName="payload-pre-processing"
Nov 02 23:55:04.892542 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:04.892518    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="05cb9af5-5f37-40e2-8a25-51702df1c674" containerName="payload-processing"
Nov 02 23:55:04.896077 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:04.896059    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"
Nov 02 23:55:04.945597 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:04.945563    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-a8a1dd-serving-cert\""
Nov 02 23:55:04.947442 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:04.947413    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"]
Nov 02 23:55:04.972108 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:04.972073    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/850d1e3e-f407-4294-9e9a-61471e09b21b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb\" (UID: \"850d1e3e-f407-4294-9e9a-61471e09b21b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"
Nov 02 23:55:04.972309 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:04.972129    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/850d1e3e-f407-4294-9e9a-61471e09b21b-metrics-certs\") pod \"maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb\" (UID: \"850d1e3e-f407-4294-9e9a-61471e09b21b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"
Nov 02 23:55:04.972382 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:04.972297    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/850d1e3e-f407-4294-9e9a-61471e09b21b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb\" (UID: \"850d1e3e-f407-4294-9e9a-61471e09b21b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"
Nov 02 23:55:04.972447 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:04.972404    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96bp7\" (UniqueName: \"kubernetes.io/projected/850d1e3e-f407-4294-9e9a-61471e09b21b-kube-api-access-96bp7\") pod \"maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb\" (UID: \"850d1e3e-f407-4294-9e9a-61471e09b21b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"
Nov 02 23:55:05.073547 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:05.073508    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96bp7\" (UniqueName: \"kubernetes.io/projected/850d1e3e-f407-4294-9e9a-61471e09b21b-kube-api-access-96bp7\") pod \"maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb\" (UID: \"850d1e3e-f407-4294-9e9a-61471e09b21b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"
Nov 02 23:55:05.073733 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:05.073610    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/850d1e3e-f407-4294-9e9a-61471e09b21b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb\" (UID: \"850d1e3e-f407-4294-9e9a-61471e09b21b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"
Nov 02 23:55:05.073733 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:05.073655    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/850d1e3e-f407-4294-9e9a-61471e09b21b-metrics-certs\") pod \"maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb\" (UID: \"850d1e3e-f407-4294-9e9a-61471e09b21b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"
Nov 02 23:55:05.073859 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:05.073745    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/850d1e3e-f407-4294-9e9a-61471e09b21b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb\" (UID: \"850d1e3e-f407-4294-9e9a-61471e09b21b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"
Nov 02 23:55:05.073859 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:55:05.073762    4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-serving-cert: secret "maas-api-e2e-extmod-b-a8a1dd-serving-cert" not found
Nov 02 23:55:05.073859 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:55:05.073840    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/850d1e3e-f407-4294-9e9a-61471e09b21b-maas-api-tls podName:850d1e3e-f407-4294-9e9a-61471e09b21b nodeName:}" failed. No retries permitted until 2026-11-02 23:55:05.573821858 +0000 UTC m=+3745.142176629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/850d1e3e-f407-4294-9e9a-61471e09b21b-maas-api-tls") pod "maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb" (UID: "850d1e3e-f407-4294-9e9a-61471e09b21b") : secret "maas-api-e2e-extmod-b-a8a1dd-serving-cert" not found
Nov 02 23:55:05.074481 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:05.074457    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/850d1e3e-f407-4294-9e9a-61471e09b21b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb\" (UID: \"850d1e3e-f407-4294-9e9a-61471e09b21b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"
Nov 02 23:55:05.076153 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:05.076131    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/850d1e3e-f407-4294-9e9a-61471e09b21b-metrics-certs\") pod \"maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb\" (UID: \"850d1e3e-f407-4294-9e9a-61471e09b21b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"
Nov 02 23:55:05.111153 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:05.111118    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:05.128639 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:05.128612    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96bp7\" (UniqueName: \"kubernetes.io/projected/850d1e3e-f407-4294-9e9a-61471e09b21b-kube-api-access-96bp7\") pod \"maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb\" (UID: \"850d1e3e-f407-4294-9e9a-61471e09b21b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"
Nov 02 23:55:05.274555 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:05.274513    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:05.432328 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:05.432290    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:05.579027 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:05.578919    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/850d1e3e-f407-4294-9e9a-61471e09b21b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb\" (UID: \"850d1e3e-f407-4294-9e9a-61471e09b21b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"
Nov 02 23:55:05.581490 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:05.581458    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/850d1e3e-f407-4294-9e9a-61471e09b21b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb\" (UID: \"850d1e3e-f407-4294-9e9a-61471e09b21b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"
Nov 02 23:55:05.819096 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:05.819063    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"
Nov 02 23:55:06.188363 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:06.188334    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"]
Nov 02 23:55:06.188735 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:55:06.188487    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850d1e3e_f407_4294_9e9a_61471e09b21b.slice/crio-19cc4bb2c46cf4e1ce02577f1319b746890d2fa459f494174f03146475136880 WatchSource:0}: Error finding container 19cc4bb2c46cf4e1ce02577f1319b746890d2fa459f494174f03146475136880: Status 404 returned error can't find the container with id 19cc4bb2c46cf4e1ce02577f1319b746890d2fa459f494174f03146475136880
Nov 02 23:55:06.190120 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:06.190102    4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 23:55:06.322914 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:06.322876    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb" event={"ID":"850d1e3e-f407-4294-9e9a-61471e09b21b","Type":"ContainerStarted","Data":"19cc4bb2c46cf4e1ce02577f1319b746890d2fa459f494174f03146475136880"}
Nov 02 23:55:07.330682 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:07.330575    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb" event={"ID":"850d1e3e-f407-4294-9e9a-61471e09b21b","Type":"ContainerStarted","Data":"77fdd599357455fe973fddefc55eb3e4fc2d7f11daefed81d0f6ff7380e2ebeb"}
Nov 02 23:55:07.330682 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:07.330647    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"
Nov 02 23:55:07.441533 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:07.441481    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb" podStartSLOduration=2.570612267 podStartE2EDuration="3.441464967s" totalImagesPullingTime="870.8527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:55:07.439131038 +0000 UTC m=+3747.007485825" watchObservedRunningTime="2026-11-02 23:55:07.441464967 +0000 UTC m=+3747.009819811"
Nov 02 23:55:09.854014 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:09.853959    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:10.747673 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:10.747632    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:11.751198 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:55:11.751158    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:55:13.262568 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:13.262540    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:13.342625 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:13.342599    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"
Nov 02 23:55:13.417933 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:13.417900    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:14.384136 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:14.384093    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:16.041471 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:16.041437    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw"]
Nov 02 23:55:16.144441 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:16.144406    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z"]
Nov 02 23:55:16.429745 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:16.429663    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:16.656962 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:16.656930    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:16.858763 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:16.858733    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:19.062673 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:19.062626    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:21.042599 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.042563    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw" podUID="e1728df9-e8b4-4c64-adc6-9178bdd638d8" containerName="payload-processing" containerID="cri-o://f7915e6d7cdb6b54717f695f9fcd8071ff31168e1ab827f6542cf03ed22dcacf" gracePeriod=25
Nov 02 23:55:21.145310 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.145269    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z" podUID="cfca5461-e8a7-4538-9e13-26aeca11ed47" containerName="payload-pre-processing" containerID="cri-o://4597d2dfc0cf0e2eb801e05cf5138747558b38ebeec98bcc87e547d6285d3148" gracePeriod=25
Nov 02 23:55:21.307602 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.307532    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw"
Nov 02 23:55:21.363353 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.363323    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1728df9-e8b4-4c64-adc6-9178bdd638d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1728df9-e8b4-4c64-adc6-9178bdd638d8-plugins-config-volume\") pod \"e1728df9-e8b4-4c64-adc6-9178bdd638d8\" (UID: \"e1728df9-e8b4-4c64-adc6-9178bdd638d8\") "
Nov 02 23:55:21.363549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.363485    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1728df9-e8b4-4c64-adc6-9178bdd638d8-kube-api-access-j4x5x\" (UniqueName: \"kubernetes.io/projected/e1728df9-e8b4-4c64-adc6-9178bdd638d8-kube-api-access-j4x5x\") pod \"e1728df9-e8b4-4c64-adc6-9178bdd638d8\" (UID: \"e1728df9-e8b4-4c64-adc6-9178bdd638d8\") "
Nov 02 23:55:21.363810 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.363766    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1728df9-e8b4-4c64-adc6-9178bdd638d8-plugins-config-volume" pod "e1728df9-e8b4-4c64-adc6-9178bdd638d8" (UID: "e1728df9-e8b4-4c64-adc6-9178bdd638d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:55:21.364048 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.364027    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1728df9-e8b4-4c64-adc6-9178bdd638d8-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:55:21.365656 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.365629    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1728df9-e8b4-4c64-adc6-9178bdd638d8-kube-api-access-j4x5x" pod "e1728df9-e8b4-4c64-adc6-9178bdd638d8" (UID: "e1728df9-e8b4-4c64-adc6-9178bdd638d8"). InnerVolumeSpecName "kube-api-access-j4x5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:55:21.411315 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.411295    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z"
Nov 02 23:55:21.429116 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.429086    4294 generic.go:356] "Generic (PLEG): container finished" podID="cfca5461-e8a7-4538-9e13-26aeca11ed47" containerID="4597d2dfc0cf0e2eb801e05cf5138747558b38ebeec98bcc87e547d6285d3148" exitCode=0
Nov 02 23:55:21.429280 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.429144    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z"
Nov 02 23:55:21.429280 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.429169    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z" event={"ID":"cfca5461-e8a7-4538-9e13-26aeca11ed47","Type":"ContainerDied","Data":"4597d2dfc0cf0e2eb801e05cf5138747558b38ebeec98bcc87e547d6285d3148"}
Nov 02 23:55:21.429280 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.429225    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z" event={"ID":"cfca5461-e8a7-4538-9e13-26aeca11ed47","Type":"ContainerDied","Data":"be1629e64df98f5aad4516f87961632f75837ac96ff617a5bf2ddc42c4c2a94a"}
Nov 02 23:55:21.429280 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.429253    4294 scope.go:122] "RemoveContainer" containerID="4597d2dfc0cf0e2eb801e05cf5138747558b38ebeec98bcc87e547d6285d3148"
Nov 02 23:55:21.430727 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.430706    4294 generic.go:356] "Generic (PLEG): container finished" podID="e1728df9-e8b4-4c64-adc6-9178bdd638d8" containerID="f7915e6d7cdb6b54717f695f9fcd8071ff31168e1ab827f6542cf03ed22dcacf" exitCode=0
Nov 02 23:55:21.430839 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.430750    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw" event={"ID":"e1728df9-e8b4-4c64-adc6-9178bdd638d8","Type":"ContainerDied","Data":"f7915e6d7cdb6b54717f695f9fcd8071ff31168e1ab827f6542cf03ed22dcacf"}
Nov 02 23:55:21.430839 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.430795    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw" event={"ID":"e1728df9-e8b4-4c64-adc6-9178bdd638d8","Type":"ContainerDied","Data":"39568e187638f622575c87d17e4e2c0fae6cdea08c08b0fe5e46697c6565b57d"}
Nov 02 23:55:21.430839 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.430758    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw"
Nov 02 23:55:21.443864 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.443844    4294 scope.go:122] "RemoveContainer" containerID="4597d2dfc0cf0e2eb801e05cf5138747558b38ebeec98bcc87e547d6285d3148"
Nov 02 23:55:21.444187 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:55:21.444170    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4597d2dfc0cf0e2eb801e05cf5138747558b38ebeec98bcc87e547d6285d3148\": container with ID starting with 4597d2dfc0cf0e2eb801e05cf5138747558b38ebeec98bcc87e547d6285d3148 not found: ID does not exist" containerID="4597d2dfc0cf0e2eb801e05cf5138747558b38ebeec98bcc87e547d6285d3148"
Nov 02 23:55:21.444253 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.444194    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4597d2dfc0cf0e2eb801e05cf5138747558b38ebeec98bcc87e547d6285d3148"} err="failed to get container status \"4597d2dfc0cf0e2eb801e05cf5138747558b38ebeec98bcc87e547d6285d3148\": rpc error: code = NotFound desc = could not find container \"4597d2dfc0cf0e2eb801e05cf5138747558b38ebeec98bcc87e547d6285d3148\": container with ID starting with 4597d2dfc0cf0e2eb801e05cf5138747558b38ebeec98bcc87e547d6285d3148 not found: ID does not exist"
Nov 02 23:55:21.444253 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.444216    4294 scope.go:122] "RemoveContainer" containerID="f7915e6d7cdb6b54717f695f9fcd8071ff31168e1ab827f6542cf03ed22dcacf"
Nov 02 23:55:21.453790 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.453768    4294 scope.go:122] "RemoveContainer" containerID="f7915e6d7cdb6b54717f695f9fcd8071ff31168e1ab827f6542cf03ed22dcacf"
Nov 02 23:55:21.454099 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:55:21.454080    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7915e6d7cdb6b54717f695f9fcd8071ff31168e1ab827f6542cf03ed22dcacf\": container with ID starting with f7915e6d7cdb6b54717f695f9fcd8071ff31168e1ab827f6542cf03ed22dcacf not found: ID does not exist" containerID="f7915e6d7cdb6b54717f695f9fcd8071ff31168e1ab827f6542cf03ed22dcacf"
Nov 02 23:55:21.454198 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.454103    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7915e6d7cdb6b54717f695f9fcd8071ff31168e1ab827f6542cf03ed22dcacf"} err="failed to get container status \"f7915e6d7cdb6b54717f695f9fcd8071ff31168e1ab827f6542cf03ed22dcacf\": rpc error: code = NotFound desc = could not find container \"f7915e6d7cdb6b54717f695f9fcd8071ff31168e1ab827f6542cf03ed22dcacf\": container with ID starting with f7915e6d7cdb6b54717f695f9fcd8071ff31168e1ab827f6542cf03ed22dcacf not found: ID does not exist"
Nov 02 23:55:21.464658 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.464610    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfca5461-e8a7-4538-9e13-26aeca11ed47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfca5461-e8a7-4538-9e13-26aeca11ed47-plugins-config-volume\") pod \"cfca5461-e8a7-4538-9e13-26aeca11ed47\" (UID: \"cfca5461-e8a7-4538-9e13-26aeca11ed47\") "
Nov 02 23:55:21.464795 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.464733    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfca5461-e8a7-4538-9e13-26aeca11ed47-kube-api-access-b6pf9\" (UniqueName: \"kubernetes.io/projected/cfca5461-e8a7-4538-9e13-26aeca11ed47-kube-api-access-b6pf9\") pod \"cfca5461-e8a7-4538-9e13-26aeca11ed47\" (UID: \"cfca5461-e8a7-4538-9e13-26aeca11ed47\") "
Nov 02 23:55:21.465025 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.465002    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfca5461-e8a7-4538-9e13-26aeca11ed47-plugins-config-volume" pod "cfca5461-e8a7-4538-9e13-26aeca11ed47" (UID: "cfca5461-e8a7-4538-9e13-26aeca11ed47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:55:21.465216 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.465199    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4x5x\" (UniqueName: \"kubernetes.io/projected/e1728df9-e8b4-4c64-adc6-9178bdd638d8-kube-api-access-j4x5x\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:55:21.465307 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.465218    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfca5461-e8a7-4538-9e13-26aeca11ed47-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:55:21.466791 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.466771    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfca5461-e8a7-4538-9e13-26aeca11ed47-kube-api-access-b6pf9" pod "cfca5461-e8a7-4538-9e13-26aeca11ed47" (UID: "cfca5461-e8a7-4538-9e13-26aeca11ed47"). InnerVolumeSpecName "kube-api-access-b6pf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:55:21.566369 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.566283    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6pf9\" (UniqueName: \"kubernetes.io/projected/cfca5461-e8a7-4538-9e13-26aeca11ed47-kube-api-access-b6pf9\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:55:21.630975 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.630932    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw"]
Nov 02 23:55:21.634173 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.634152    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3bcb9f-5f456b8f4f-9zqgw"]
Nov 02 23:55:21.845338 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.845255    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z"]
Nov 02 23:55:21.850858 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:21.850830    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3bcb9f-9b44f8847-mq45z"]
Nov 02 23:55:22.548067 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:22.548031    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfca5461-e8a7-4538-9e13-26aeca11ed47" path="/var/lib/kubelet/pods/cfca5461-e8a7-4538-9e13-26aeca11ed47/volumes"
Nov 02 23:55:22.548482 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:22.548469    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1728df9-e8b4-4c64-adc6-9178bdd638d8" path="/var/lib/kubelet/pods/e1728df9-e8b4-4c64-adc6-9178bdd638d8/volumes"
Nov 02 23:55:22.554564 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:22.554541    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:23.981991 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:23.981936    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:28.657061 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:28.657029    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:31.014801 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:31.014770    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:33.161545 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:33.161501    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:36.981902 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:36.981869    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:37.673147 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:37.673109    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99"]
Nov 02 23:55:37.722861 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:37.722784    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:38.040484 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:38.040447    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:38.200433 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:38.200403    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:41.750144 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:55:41.750099    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:55:42.647757 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:42.647719    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:42.674066 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:42.673965    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99" podUID="25addf9e-2173-4971-9112-02e9857a65d4" containerName="payload-processing" containerID="cri-o://fafd33d687b5dc874eb61bdfc3a52aa6f11982f5de45fcfde8b87876434d66c4" gracePeriod=25
Nov 02 23:55:42.813957 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:42.813923    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:42.940393 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:42.940366    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99"
Nov 02 23:55:42.973512 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:42.973481    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:43.122012 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:43.121942    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25addf9e-2173-4971-9112-02e9857a65d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25addf9e-2173-4971-9112-02e9857a65d4-plugins-config-volume\") pod \"25addf9e-2173-4971-9112-02e9857a65d4\" (UID: \"25addf9e-2173-4971-9112-02e9857a65d4\") "
Nov 02 23:55:43.122012 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:43.122014    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25addf9e-2173-4971-9112-02e9857a65d4-kube-api-access-6tmqs\" (UniqueName: \"kubernetes.io/projected/25addf9e-2173-4971-9112-02e9857a65d4-kube-api-access-6tmqs\") pod \"25addf9e-2173-4971-9112-02e9857a65d4\" (UID: \"25addf9e-2173-4971-9112-02e9857a65d4\") "
Nov 02 23:55:43.122349 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:43.122320    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25addf9e-2173-4971-9112-02e9857a65d4-plugins-config-volume" pod "25addf9e-2173-4971-9112-02e9857a65d4" (UID: "25addf9e-2173-4971-9112-02e9857a65d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:55:43.122614 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:43.122594    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25addf9e-2173-4971-9112-02e9857a65d4-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:55:43.124159 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:43.124135    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25addf9e-2173-4971-9112-02e9857a65d4-kube-api-access-6tmqs" pod "25addf9e-2173-4971-9112-02e9857a65d4" (UID: "25addf9e-2173-4971-9112-02e9857a65d4"). InnerVolumeSpecName "kube-api-access-6tmqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:55:43.224533 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:43.224444    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tmqs\" (UniqueName: \"kubernetes.io/projected/25addf9e-2173-4971-9112-02e9857a65d4-kube-api-access-6tmqs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:55:43.573626 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:43.573591    4294 generic.go:356] "Generic (PLEG): container finished" podID="25addf9e-2173-4971-9112-02e9857a65d4" containerID="fafd33d687b5dc874eb61bdfc3a52aa6f11982f5de45fcfde8b87876434d66c4" exitCode=0
Nov 02 23:55:43.573846 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:43.573635    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99"
Nov 02 23:55:43.573846 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:43.573672    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99" event={"ID":"25addf9e-2173-4971-9112-02e9857a65d4","Type":"ContainerDied","Data":"fafd33d687b5dc874eb61bdfc3a52aa6f11982f5de45fcfde8b87876434d66c4"}
Nov 02 23:55:43.573846 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:43.573733    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99" event={"ID":"25addf9e-2173-4971-9112-02e9857a65d4","Type":"ContainerDied","Data":"e69b24c293dceb2d1debfc745315bd8852462ff37e5e910a370192a9c02a7624"}
Nov 02 23:55:43.573846 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:43.573765    4294 scope.go:122] "RemoveContainer" containerID="fafd33d687b5dc874eb61bdfc3a52aa6f11982f5de45fcfde8b87876434d66c4"
Nov 02 23:55:43.584634 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:43.584616    4294 scope.go:122] "RemoveContainer" containerID="fafd33d687b5dc874eb61bdfc3a52aa6f11982f5de45fcfde8b87876434d66c4"
Nov 02 23:55:43.584908 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:55:43.584889    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fafd33d687b5dc874eb61bdfc3a52aa6f11982f5de45fcfde8b87876434d66c4\": container with ID starting with fafd33d687b5dc874eb61bdfc3a52aa6f11982f5de45fcfde8b87876434d66c4 not found: ID does not exist" containerID="fafd33d687b5dc874eb61bdfc3a52aa6f11982f5de45fcfde8b87876434d66c4"
Nov 02 23:55:43.584992 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:43.584919    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fafd33d687b5dc874eb61bdfc3a52aa6f11982f5de45fcfde8b87876434d66c4"} err="failed to get container status \"fafd33d687b5dc874eb61bdfc3a52aa6f11982f5de45fcfde8b87876434d66c4\": rpc error: code = NotFound desc = could not find container \"fafd33d687b5dc874eb61bdfc3a52aa6f11982f5de45fcfde8b87876434d66c4\": container with ID starting with fafd33d687b5dc874eb61bdfc3a52aa6f11982f5de45fcfde8b87876434d66c4 not found: ID does not exist"
Nov 02 23:55:43.731677 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:43.731637    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99"]
Nov 02 23:55:43.778697 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:43.778662    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8c2bff-9f99ff889-zgp99"]
Nov 02 23:55:44.548498 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:44.548464    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25addf9e-2173-4971-9112-02e9857a65d4" path="/var/lib/kubelet/pods/25addf9e-2173-4971-9112-02e9857a65d4/volumes"
Nov 02 23:55:47.496594 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:47.496560    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:52.502929 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:52.502886    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:55:57.092816 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:55:57.092780    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:01.648019 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:01.647963    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:06.347997 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:06.347949    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:10.818734 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:10.818692    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:11.751729 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:56:11.751686    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:56:15.394597 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:15.394562    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:20.173256 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:20.173217    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:24.859803 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:24.859720    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:29.537224 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:29.537192    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:32.586000 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:32.585950    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:33.483020 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:33.482958    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:35.126843 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:35.126808    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:37.788064 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:37.788027    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:37.958996 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:37.958954    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:38.571610 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:38.571577    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:39.311697 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:39.311658    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:39.843670 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:39.843632    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:40.014371 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:40.014333    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:40.691065 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:40.691030    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:41.751137 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:56:41.751097    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:56:41.901033 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:41.900956    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk"]
Nov 02 23:56:42.001566 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:42.001529    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9"]
Nov 02 23:56:42.274922 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:42.274878    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:42.433804 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:42.433758    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:43.214808 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:43.214772    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:43.372426 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:43.372388    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:44.887493 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:44.887456    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:46.545804 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:46.545759    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:46.901450 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:46.901351    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk" podUID="33dd7ffb-c342-476c-8ab6-6370fff64b9b" containerName="payload-processing" containerID="cri-o://64bab326a8c42e05d3c0fd0fdef7294659ca9a18d7247e3e52cb810ae0613282" gracePeriod=25
Nov 02 23:56:47.001838 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.001800    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9" podUID="24256826-ee09-4791-af71-55d2e820dfdc" containerName="payload-pre-processing" containerID="cri-o://0557518c82cc5e9ba825e00ac93eb0c465ba992eab6efe167ee331ff7df1423e" gracePeriod=25
Nov 02 23:56:47.160073 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.159475    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk"
Nov 02 23:56:47.261116 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.261085    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33dd7ffb-c342-476c-8ab6-6370fff64b9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33dd7ffb-c342-476c-8ab6-6370fff64b9b-plugins-config-volume\") pod \"33dd7ffb-c342-476c-8ab6-6370fff64b9b\" (UID: \"33dd7ffb-c342-476c-8ab6-6370fff64b9b\") "
Nov 02 23:56:47.261297 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.261129    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33dd7ffb-c342-476c-8ab6-6370fff64b9b-kube-api-access-42xzz\" (UniqueName: \"kubernetes.io/projected/33dd7ffb-c342-476c-8ab6-6370fff64b9b-kube-api-access-42xzz\") pod \"33dd7ffb-c342-476c-8ab6-6370fff64b9b\" (UID: \"33dd7ffb-c342-476c-8ab6-6370fff64b9b\") "
Nov 02 23:56:47.261496 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.261468    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33dd7ffb-c342-476c-8ab6-6370fff64b9b-plugins-config-volume" pod "33dd7ffb-c342-476c-8ab6-6370fff64b9b" (UID: "33dd7ffb-c342-476c-8ab6-6370fff64b9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:56:47.261745 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.261725    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33dd7ffb-c342-476c-8ab6-6370fff64b9b-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:56:47.263322 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.263299    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33dd7ffb-c342-476c-8ab6-6370fff64b9b-kube-api-access-42xzz" pod "33dd7ffb-c342-476c-8ab6-6370fff64b9b" (UID: "33dd7ffb-c342-476c-8ab6-6370fff64b9b"). InnerVolumeSpecName "kube-api-access-42xzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:56:47.287525 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.287500    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9"
Nov 02 23:56:47.362432 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.362407    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24256826-ee09-4791-af71-55d2e820dfdc-kube-api-access-4kg4r\" (UniqueName: \"kubernetes.io/projected/24256826-ee09-4791-af71-55d2e820dfdc-kube-api-access-4kg4r\") pod \"24256826-ee09-4791-af71-55d2e820dfdc\" (UID: \"24256826-ee09-4791-af71-55d2e820dfdc\") "
Nov 02 23:56:47.362619 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.362479    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24256826-ee09-4791-af71-55d2e820dfdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24256826-ee09-4791-af71-55d2e820dfdc-plugins-config-volume\") pod \"24256826-ee09-4791-af71-55d2e820dfdc\" (UID: \"24256826-ee09-4791-af71-55d2e820dfdc\") "
Nov 02 23:56:47.362840 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.362817    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24256826-ee09-4791-af71-55d2e820dfdc-plugins-config-volume" pod "24256826-ee09-4791-af71-55d2e820dfdc" (UID: "24256826-ee09-4791-af71-55d2e820dfdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:56:47.362926 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.362894    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42xzz\" (UniqueName: \"kubernetes.io/projected/33dd7ffb-c342-476c-8ab6-6370fff64b9b-kube-api-access-42xzz\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:56:47.364675 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.364647    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24256826-ee09-4791-af71-55d2e820dfdc-kube-api-access-4kg4r" pod "24256826-ee09-4791-af71-55d2e820dfdc" (UID: "24256826-ee09-4791-af71-55d2e820dfdc"). InnerVolumeSpecName "kube-api-access-4kg4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:56:47.464261 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.464172    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24256826-ee09-4791-af71-55d2e820dfdc-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:56:47.464261 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.464204    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kg4r\" (UniqueName: \"kubernetes.io/projected/24256826-ee09-4791-af71-55d2e820dfdc-kube-api-access-4kg4r\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:56:47.994273 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.994237    4294 generic.go:356] "Generic (PLEG): container finished" podID="24256826-ee09-4791-af71-55d2e820dfdc" containerID="0557518c82cc5e9ba825e00ac93eb0c465ba992eab6efe167ee331ff7df1423e" exitCode=0
Nov 02 23:56:47.994781 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.994286    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9"
Nov 02 23:56:47.994781 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.994325    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9" event={"ID":"24256826-ee09-4791-af71-55d2e820dfdc","Type":"ContainerDied","Data":"0557518c82cc5e9ba825e00ac93eb0c465ba992eab6efe167ee331ff7df1423e"}
Nov 02 23:56:47.994781 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.994377    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9" event={"ID":"24256826-ee09-4791-af71-55d2e820dfdc","Type":"ContainerDied","Data":"aeba83e199b5c7b46ad7fa998305fb61de194b4def4ff5cfa06210cdc342d8c2"}
Nov 02 23:56:47.994781 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.994400    4294 scope.go:122] "RemoveContainer" containerID="0557518c82cc5e9ba825e00ac93eb0c465ba992eab6efe167ee331ff7df1423e"
Nov 02 23:56:47.995855 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.995837    4294 generic.go:356] "Generic (PLEG): container finished" podID="33dd7ffb-c342-476c-8ab6-6370fff64b9b" containerID="64bab326a8c42e05d3c0fd0fdef7294659ca9a18d7247e3e52cb810ae0613282" exitCode=0
Nov 02 23:56:47.995934 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.995879    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk"
Nov 02 23:56:47.995934 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.995899    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk" event={"ID":"33dd7ffb-c342-476c-8ab6-6370fff64b9b","Type":"ContainerDied","Data":"64bab326a8c42e05d3c0fd0fdef7294659ca9a18d7247e3e52cb810ae0613282"}
Nov 02 23:56:47.996082 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:47.995959    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk" event={"ID":"33dd7ffb-c342-476c-8ab6-6370fff64b9b","Type":"ContainerDied","Data":"da6923f1f29b63c1fb7dfe3e642d94ae9889a5df200e43ea320a50f1fab0931d"}
Nov 02 23:56:48.006364 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:48.006341    4294 scope.go:122] "RemoveContainer" containerID="0557518c82cc5e9ba825e00ac93eb0c465ba992eab6efe167ee331ff7df1423e"
Nov 02 23:56:48.006680 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:56:48.006649    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0557518c82cc5e9ba825e00ac93eb0c465ba992eab6efe167ee331ff7df1423e\": container with ID starting with 0557518c82cc5e9ba825e00ac93eb0c465ba992eab6efe167ee331ff7df1423e not found: ID does not exist" containerID="0557518c82cc5e9ba825e00ac93eb0c465ba992eab6efe167ee331ff7df1423e"
Nov 02 23:56:48.006748 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:48.006679    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0557518c82cc5e9ba825e00ac93eb0c465ba992eab6efe167ee331ff7df1423e"} err="failed to get container status \"0557518c82cc5e9ba825e00ac93eb0c465ba992eab6efe167ee331ff7df1423e\": rpc error: code = NotFound desc = could not find container \"0557518c82cc5e9ba825e00ac93eb0c465ba992eab6efe167ee331ff7df1423e\": container with ID starting with 0557518c82cc5e9ba825e00ac93eb0c465ba992eab6efe167ee331ff7df1423e not found: ID does not exist"
Nov 02 23:56:48.006748 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:48.006695    4294 scope.go:122] "RemoveContainer" containerID="64bab326a8c42e05d3c0fd0fdef7294659ca9a18d7247e3e52cb810ae0613282"
Nov 02 23:56:48.016801 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:48.016785    4294 scope.go:122] "RemoveContainer" containerID="64bab326a8c42e05d3c0fd0fdef7294659ca9a18d7247e3e52cb810ae0613282"
Nov 02 23:56:48.017066 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:56:48.017046    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64bab326a8c42e05d3c0fd0fdef7294659ca9a18d7247e3e52cb810ae0613282\": container with ID starting with 64bab326a8c42e05d3c0fd0fdef7294659ca9a18d7247e3e52cb810ae0613282 not found: ID does not exist" containerID="64bab326a8c42e05d3c0fd0fdef7294659ca9a18d7247e3e52cb810ae0613282"
Nov 02 23:56:48.017132 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:48.017080    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64bab326a8c42e05d3c0fd0fdef7294659ca9a18d7247e3e52cb810ae0613282"} err="failed to get container status \"64bab326a8c42e05d3c0fd0fdef7294659ca9a18d7247e3e52cb810ae0613282\": rpc error: code = NotFound desc = could not find container \"64bab326a8c42e05d3c0fd0fdef7294659ca9a18d7247e3e52cb810ae0613282\": container with ID starting with 64bab326a8c42e05d3c0fd0fdef7294659ca9a18d7247e3e52cb810ae0613282 not found: ID does not exist"
Nov 02 23:56:48.152207 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:48.152175    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk"]
Nov 02 23:56:48.155376 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:48.155350    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-e65845-5d44465b89-rk5nk"]
Nov 02 23:56:48.259014 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:48.258962    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9"]
Nov 02 23:56:48.263487 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:48.263464    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e65845-7c8f9c9ddc-jhcl9"]
Nov 02 23:56:48.548637 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:48.548555    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24256826-ee09-4791-af71-55d2e820dfdc" path="/var/lib/kubelet/pods/24256826-ee09-4791-af71-55d2e820dfdc/volumes"
Nov 02 23:56:48.549057 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:48.549038    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33dd7ffb-c342-476c-8ab6-6370fff64b9b" path="/var/lib/kubelet/pods/33dd7ffb-c342-476c-8ab6-6370fff64b9b/volumes"
Nov 02 23:56:49.023046 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:49.023016    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"]
Nov 02 23:56:49.023488 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:49.023205    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p" podUID="d129273b-8713-4065-b518-7001a19aefd1" containerName="maas-api" containerID="cri-o://4fd25e91763332fb65da0a08abcfa36a4f70e4f333690f706d870d3fea857259" gracePeriod=30
Nov 02 23:56:49.144965 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:49.144912    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:49.293689 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:49.293666    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"
Nov 02 23:56:49.386764 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:49.386732    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d129273b-8713-4065-b518-7001a19aefd1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d129273b-8713-4065-b518-7001a19aefd1-metrics-certs\") pod \"d129273b-8713-4065-b518-7001a19aefd1\" (UID: \"d129273b-8713-4065-b518-7001a19aefd1\") "
Nov 02 23:56:49.387060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:49.386836    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d129273b-8713-4065-b518-7001a19aefd1-kube-api-access-vg4hd\" (UniqueName: \"kubernetes.io/projected/d129273b-8713-4065-b518-7001a19aefd1-kube-api-access-vg4hd\") pod \"d129273b-8713-4065-b518-7001a19aefd1\" (UID: \"d129273b-8713-4065-b518-7001a19aefd1\") "
Nov 02 23:56:49.387060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:49.386950    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d129273b-8713-4065-b518-7001a19aefd1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d129273b-8713-4065-b518-7001a19aefd1-maas-api-tls\") pod \"d129273b-8713-4065-b518-7001a19aefd1\" (UID: \"d129273b-8713-4065-b518-7001a19aefd1\") "
Nov 02 23:56:49.387210 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:49.387085    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d129273b-8713-4065-b518-7001a19aefd1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d129273b-8713-4065-b518-7001a19aefd1-openshift-service-ca\") pod \"d129273b-8713-4065-b518-7001a19aefd1\" (UID: \"d129273b-8713-4065-b518-7001a19aefd1\") "
Nov 02 23:56:49.387502 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:49.387452    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d129273b-8713-4065-b518-7001a19aefd1-openshift-service-ca" pod "d129273b-8713-4065-b518-7001a19aefd1" (UID: "d129273b-8713-4065-b518-7001a19aefd1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:56:49.387636 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:49.387618    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d129273b-8713-4065-b518-7001a19aefd1-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:56:49.389018 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:49.388992    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d129273b-8713-4065-b518-7001a19aefd1-kube-api-access-vg4hd" pod "d129273b-8713-4065-b518-7001a19aefd1" (UID: "d129273b-8713-4065-b518-7001a19aefd1"). InnerVolumeSpecName "kube-api-access-vg4hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:56:49.389333 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:49.389314    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d129273b-8713-4065-b518-7001a19aefd1-maas-api-tls" pod "d129273b-8713-4065-b518-7001a19aefd1" (UID: "d129273b-8713-4065-b518-7001a19aefd1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:56:49.389419 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:49.389367    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d129273b-8713-4065-b518-7001a19aefd1-metrics-certs" pod "d129273b-8713-4065-b518-7001a19aefd1" (UID: "d129273b-8713-4065-b518-7001a19aefd1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:56:49.488670 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:49.488632    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d129273b-8713-4065-b518-7001a19aefd1-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:56:49.488670 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:49.488671    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg4hd\" (UniqueName: \"kubernetes.io/projected/d129273b-8713-4065-b518-7001a19aefd1-kube-api-access-vg4hd\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:56:49.488893 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:49.488688    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d129273b-8713-4065-b518-7001a19aefd1-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:56:49.491383 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:49.491346    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn"]
Nov 02 23:56:49.677023 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:49.676911    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:49.836723 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:49.836688    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:50.011754 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:50.011715    4294 generic.go:356] "Generic (PLEG): container finished" podID="d129273b-8713-4065-b518-7001a19aefd1" containerID="4fd25e91763332fb65da0a08abcfa36a4f70e4f333690f706d870d3fea857259" exitCode=0
Nov 02 23:56:50.011954 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:50.011766    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"
Nov 02 23:56:50.011954 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:50.011779    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p" event={"ID":"d129273b-8713-4065-b518-7001a19aefd1","Type":"ContainerDied","Data":"4fd25e91763332fb65da0a08abcfa36a4f70e4f333690f706d870d3fea857259"}
Nov 02 23:56:50.011954 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:50.011833    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p" event={"ID":"d129273b-8713-4065-b518-7001a19aefd1","Type":"ContainerDied","Data":"464cf5338ec61aca95e7435c7afe4dd66d32dd5b042947b14eacec8e59b3d717"}
Nov 02 23:56:50.011954 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:50.011863    4294 scope.go:122] "RemoveContainer" containerID="4fd25e91763332fb65da0a08abcfa36a4f70e4f333690f706d870d3fea857259"
Nov 02 23:56:50.023671 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:50.023650    4294 scope.go:122] "RemoveContainer" containerID="4fd25e91763332fb65da0a08abcfa36a4f70e4f333690f706d870d3fea857259"
Nov 02 23:56:50.024033 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:56:50.023945    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd25e91763332fb65da0a08abcfa36a4f70e4f333690f706d870d3fea857259\": container with ID starting with 4fd25e91763332fb65da0a08abcfa36a4f70e4f333690f706d870d3fea857259 not found: ID does not exist" containerID="4fd25e91763332fb65da0a08abcfa36a4f70e4f333690f706d870d3fea857259"
Nov 02 23:56:50.024033 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:50.023968    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd25e91763332fb65da0a08abcfa36a4f70e4f333690f706d870d3fea857259"} err="failed to get container status \"4fd25e91763332fb65da0a08abcfa36a4f70e4f333690f706d870d3fea857259\": rpc error: code = NotFound desc = could not find container \"4fd25e91763332fb65da0a08abcfa36a4f70e4f333690f706d870d3fea857259\": container with ID starting with 4fd25e91763332fb65da0a08abcfa36a4f70e4f333690f706d870d3fea857259 not found: ID does not exist"
Nov 02 23:56:50.171176 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:50.171138    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"]
Nov 02 23:56:50.173989 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:50.173958    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-661a20-74b489ddbb-d9n2p"]
Nov 02 23:56:50.549379 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:50.549340    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d129273b-8713-4065-b518-7001a19aefd1" path="/var/lib/kubelet/pods/d129273b-8713-4065-b518-7001a19aefd1/volumes"
Nov 02 23:56:51.013180 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:51.013149    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:52.400477 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:52.400446    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:53.204552 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:53.204516    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:54.492672 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:54.492628    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn" podUID="3aaec728-9b6b-4d36-af54-3e138d400a01" containerName="payload-processing" containerID="cri-o://fa95fa983252d97fa2b39223928d3d877f6f4fea1ac0207777174e317177cceb" gracePeriod=25
Nov 02 23:56:54.765952 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:54.765890    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn"
Nov 02 23:56:54.849318 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:54.849289    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aaec728-9b6b-4d36-af54-3e138d400a01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aaec728-9b6b-4d36-af54-3e138d400a01-plugins-config-volume\") pod \"3aaec728-9b6b-4d36-af54-3e138d400a01\" (UID: \"3aaec728-9b6b-4d36-af54-3e138d400a01\") "
Nov 02 23:56:54.849512 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:54.849488    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aaec728-9b6b-4d36-af54-3e138d400a01-kube-api-access-w9grb\" (UniqueName: \"kubernetes.io/projected/3aaec728-9b6b-4d36-af54-3e138d400a01-kube-api-access-w9grb\") pod \"3aaec728-9b6b-4d36-af54-3e138d400a01\" (UID: \"3aaec728-9b6b-4d36-af54-3e138d400a01\") "
Nov 02 23:56:54.849680 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:54.849656    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aaec728-9b6b-4d36-af54-3e138d400a01-plugins-config-volume" pod "3aaec728-9b6b-4d36-af54-3e138d400a01" (UID: "3aaec728-9b6b-4d36-af54-3e138d400a01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:56:54.849856 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:54.849841    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aaec728-9b6b-4d36-af54-3e138d400a01-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:56:54.851750 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:54.851721    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aaec728-9b6b-4d36-af54-3e138d400a01-kube-api-access-w9grb" pod "3aaec728-9b6b-4d36-af54-3e138d400a01" (UID: "3aaec728-9b6b-4d36-af54-3e138d400a01"). InnerVolumeSpecName "kube-api-access-w9grb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:56:54.950778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:54.950745    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9grb\" (UniqueName: \"kubernetes.io/projected/3aaec728-9b6b-4d36-af54-3e138d400a01-kube-api-access-w9grb\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:56:55.046748 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:55.046662    4294 generic.go:356] "Generic (PLEG): container finished" podID="3aaec728-9b6b-4d36-af54-3e138d400a01" containerID="fa95fa983252d97fa2b39223928d3d877f6f4fea1ac0207777174e317177cceb" exitCode=0
Nov 02 23:56:55.046748 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:55.046709    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn"
Nov 02 23:56:55.046936 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:55.046747    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn" event={"ID":"3aaec728-9b6b-4d36-af54-3e138d400a01","Type":"ContainerDied","Data":"fa95fa983252d97fa2b39223928d3d877f6f4fea1ac0207777174e317177cceb"}
Nov 02 23:56:55.046936 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:55.046807    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn" event={"ID":"3aaec728-9b6b-4d36-af54-3e138d400a01","Type":"ContainerDied","Data":"173d688e76d74ccb816175a0fa3735272d8b8545022e03332d9dd44c62c34f64"}
Nov 02 23:56:55.046936 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:55.046836    4294 scope.go:122] "RemoveContainer" containerID="fa95fa983252d97fa2b39223928d3d877f6f4fea1ac0207777174e317177cceb"
Nov 02 23:56:55.059562 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:55.059536    4294 scope.go:122] "RemoveContainer" containerID="fa95fa983252d97fa2b39223928d3d877f6f4fea1ac0207777174e317177cceb"
Nov 02 23:56:55.060059 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:56:55.060038    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa95fa983252d97fa2b39223928d3d877f6f4fea1ac0207777174e317177cceb\": container with ID starting with fa95fa983252d97fa2b39223928d3d877f6f4fea1ac0207777174e317177cceb not found: ID does not exist" containerID="fa95fa983252d97fa2b39223928d3d877f6f4fea1ac0207777174e317177cceb"
Nov 02 23:56:55.060179 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:55.060067    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa95fa983252d97fa2b39223928d3d877f6f4fea1ac0207777174e317177cceb"} err="failed to get container status \"fa95fa983252d97fa2b39223928d3d877f6f4fea1ac0207777174e317177cceb\": rpc error: code = NotFound desc = could not find container \"fa95fa983252d97fa2b39223928d3d877f6f4fea1ac0207777174e317177cceb\": container with ID starting with fa95fa983252d97fa2b39223928d3d877f6f4fea1ac0207777174e317177cceb not found: ID does not exist"
Nov 02 23:56:55.199834 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:55.199793    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:55.203867 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:55.203840    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn"]
Nov 02 23:56:55.206958 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:55.206931    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-661a20-7895974bd9-xg2tn"]
Nov 02 23:56:56.548313 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:56.548281    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aaec728-9b6b-4d36-af54-3e138d400a01" path="/var/lib/kubelet/pods/3aaec728-9b6b-4d36-af54-3e138d400a01/volumes"
Nov 02 23:56:57.100903 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:57.100862    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:57.274620 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:57.274582    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:58.230426 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:58.230393    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:58.416601 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:58.416573    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:58.662239 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:58.662209    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:59.141030 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:59.140997    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:56:59.900491 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:56:59.900461    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:00.098381 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:00.098351    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:00.463398 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:00.463368    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"]
Nov 02 23:57:00.463586 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:00.463567    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6" podUID="d1b342a5-4d24-4147-921c-21cfa6b1f9ee" containerName="istio-proxy" containerID="cri-o://5c1eec836cd021ad0fd9c5344318055060cdaa3c4db9b1f4032cdf3dea3de055" gracePeriod=30
Nov 02 23:57:00.501661 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:00.501632    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:01.943345 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:01.943309    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:02.517379 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:02.517349    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:02.675812 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:02.675780    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:05.554668 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.554638    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:05.735792 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.735727    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:57:05.884013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.883370    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-workload-certs\") pod \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") "
Nov 02 23:57:05.884013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.883439    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istiod-ca-cert\") pod \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") "
Nov 02 23:57:05.884013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.883498    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-kube-api-access-p6kp4\" (UniqueName: \"kubernetes.io/projected/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-kube-api-access-p6kp4\") pod \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") "
Nov 02 23:57:05.884013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.883541    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-token\" (UniqueName: \"kubernetes.io/projected/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-token\") pod \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") "
Nov 02 23:57:05.884013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.883632    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-data\") pod \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") "
Nov 02 23:57:05.884013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.883724    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-podinfo\") pod \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") "
Nov 02 23:57:05.884013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.883797    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-credential-socket\") pod \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") "
Nov 02 23:57:05.884013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.883848    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-envoy\") pod \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") "
Nov 02 23:57:05.884013 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.883894    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-workload-socket\") pod \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\" (UID: \"d1b342a5-4d24-4147-921c-21cfa6b1f9ee\") "
Nov 02 23:57:05.885970 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.885573    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-workload-socket" pod "d1b342a5-4d24-4147-921c-21cfa6b1f9ee" (UID: "d1b342a5-4d24-4147-921c-21cfa6b1f9ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:57:05.886347 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.886145    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-workload-certs" pod "d1b342a5-4d24-4147-921c-21cfa6b1f9ee" (UID: "d1b342a5-4d24-4147-921c-21cfa6b1f9ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:57:05.886925 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.886636    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istiod-ca-cert" pod "d1b342a5-4d24-4147-921c-21cfa6b1f9ee" (UID: "d1b342a5-4d24-4147-921c-21cfa6b1f9ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:57:05.889538 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.889502    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-kube-api-access-p6kp4" pod "d1b342a5-4d24-4147-921c-21cfa6b1f9ee" (UID: "d1b342a5-4d24-4147-921c-21cfa6b1f9ee"). InnerVolumeSpecName "kube-api-access-p6kp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:57:05.889676 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.889649    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-credential-socket" pod "d1b342a5-4d24-4147-921c-21cfa6b1f9ee" (UID: "d1b342a5-4d24-4147-921c-21cfa6b1f9ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:57:05.892296 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.892272    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-data" pod "d1b342a5-4d24-4147-921c-21cfa6b1f9ee" (UID: "d1b342a5-4d24-4147-921c-21cfa6b1f9ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:57:05.892575 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.892547    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-podinfo" pod "d1b342a5-4d24-4147-921c-21cfa6b1f9ee" (UID: "d1b342a5-4d24-4147-921c-21cfa6b1f9ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:57:05.893006 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.892824    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-envoy" pod "d1b342a5-4d24-4147-921c-21cfa6b1f9ee" (UID: "d1b342a5-4d24-4147-921c-21cfa6b1f9ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:57:05.894304 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.894283    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-token" pod "d1b342a5-4d24-4147-921c-21cfa6b1f9ee" (UID: "d1b342a5-4d24-4147-921c-21cfa6b1f9ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:57:05.987060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.986962    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:05.987060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.987016    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:05.987060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.987027    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:05.987060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.987036    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:05.987060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.987045    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6kp4\" (UniqueName: \"kubernetes.io/projected/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-kube-api-access-p6kp4\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:05.987060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.987055    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:05.987060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.987063    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:05.987402 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.987071    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:05.987402 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:05.987080    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b342a5-4d24-4147-921c-21cfa6b1f9ee-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:06.119126 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.119078    4294 generic.go:356] "Generic (PLEG): container finished" podID="d1b342a5-4d24-4147-921c-21cfa6b1f9ee" containerID="5c1eec836cd021ad0fd9c5344318055060cdaa3c4db9b1f4032cdf3dea3de055" exitCode=0
Nov 02 23:57:06.119126 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.119120    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6" event={"ID":"d1b342a5-4d24-4147-921c-21cfa6b1f9ee","Type":"ContainerDied","Data":"5c1eec836cd021ad0fd9c5344318055060cdaa3c4db9b1f4032cdf3dea3de055"}
Nov 02 23:57:06.119358 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.119163    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"
Nov 02 23:57:06.119358 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.119209    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6" event={"ID":"d1b342a5-4d24-4147-921c-21cfa6b1f9ee","Type":"ContainerDied","Data":"1444aea81ab5cf3450289c913b9dbc9f288bb08f97b74b1d8ccabde707f21382"}
Nov 02 23:57:06.119358 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.119234    4294 scope.go:122] "RemoveContainer" containerID="5c1eec836cd021ad0fd9c5344318055060cdaa3c4db9b1f4032cdf3dea3de055"
Nov 02 23:57:06.130825 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.130803    4294 scope.go:122] "RemoveContainer" containerID="9a64dafb8ff57ae4a39871ad4602482e78f8c4485f7b2848efaa92150d76f0c4"
Nov 02 23:57:06.140008 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.139993    4294 scope.go:122] "RemoveContainer" containerID="5c1eec836cd021ad0fd9c5344318055060cdaa3c4db9b1f4032cdf3dea3de055"
Nov 02 23:57:06.140263 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:57:06.140245    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c1eec836cd021ad0fd9c5344318055060cdaa3c4db9b1f4032cdf3dea3de055\": container with ID starting with 5c1eec836cd021ad0fd9c5344318055060cdaa3c4db9b1f4032cdf3dea3de055 not found: ID does not exist" containerID="5c1eec836cd021ad0fd9c5344318055060cdaa3c4db9b1f4032cdf3dea3de055"
Nov 02 23:57:06.140331 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.140270    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c1eec836cd021ad0fd9c5344318055060cdaa3c4db9b1f4032cdf3dea3de055"} err="failed to get container status \"5c1eec836cd021ad0fd9c5344318055060cdaa3c4db9b1f4032cdf3dea3de055\": rpc error: code = NotFound desc = could not find container \"5c1eec836cd021ad0fd9c5344318055060cdaa3c4db9b1f4032cdf3dea3de055\": container with ID starting with 5c1eec836cd021ad0fd9c5344318055060cdaa3c4db9b1f4032cdf3dea3de055 not found: ID does not exist"
Nov 02 23:57:06.140331 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.140290    4294 scope.go:122] "RemoveContainer" containerID="9a64dafb8ff57ae4a39871ad4602482e78f8c4485f7b2848efaa92150d76f0c4"
Nov 02 23:57:06.140499 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:57:06.140485    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a64dafb8ff57ae4a39871ad4602482e78f8c4485f7b2848efaa92150d76f0c4\": container with ID starting with 9a64dafb8ff57ae4a39871ad4602482e78f8c4485f7b2848efaa92150d76f0c4 not found: ID does not exist" containerID="9a64dafb8ff57ae4a39871ad4602482e78f8c4485f7b2848efaa92150d76f0c4"
Nov 02 23:57:06.140536 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.140503    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a64dafb8ff57ae4a39871ad4602482e78f8c4485f7b2848efaa92150d76f0c4"} err="failed to get container status \"9a64dafb8ff57ae4a39871ad4602482e78f8c4485f7b2848efaa92150d76f0c4\": rpc error: code = NotFound desc = could not find container \"9a64dafb8ff57ae4a39871ad4602482e78f8c4485f7b2848efaa92150d76f0c4\": container with ID starting with 9a64dafb8ff57ae4a39871ad4602482e78f8c4485f7b2848efaa92150d76f0c4 not found: ID does not exist"
Nov 02 23:57:06.249551 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.249517    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:06.277148 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.277118    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"]
Nov 02 23:57:06.286658 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.286639    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-661a20-openshift-default-8459b8fcb5-nvlh6"]
Nov 02 23:57:06.486836 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.486804    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:06.549114 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.548922    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1b342a5-4d24-4147-921c-21cfa6b1f9ee" path="/var/lib/kubelet/pods/d1b342a5-4d24-4147-921c-21cfa6b1f9ee/volumes"
Nov 02 23:57:06.732473 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.732444    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:06.916230 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.916144    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:06.976226 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.976186    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"]
Nov 02 23:57:06.976850 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.976830    4294 cpu_manager.go:414] "containerMap: removing container" podUID="d129273b-8713-4065-b518-7001a19aefd1" containerName="maas-api"
Nov 02 23:57:06.976850 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.976850    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d129273b-8713-4065-b518-7001a19aefd1" containerName="maas-api"
Nov 02 23:57:06.977087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.976865    4294 cpu_manager.go:414] "containerMap: removing container" podUID="cfca5461-e8a7-4538-9e13-26aeca11ed47" containerName="payload-pre-processing"
Nov 02 23:57:06.977087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.976872    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfca5461-e8a7-4538-9e13-26aeca11ed47" containerName="payload-pre-processing"
Nov 02 23:57:06.977087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.976893    4294 cpu_manager.go:414] "containerMap: removing container" podUID="3aaec728-9b6b-4d36-af54-3e138d400a01" containerName="payload-processing"
Nov 02 23:57:06.977087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.976900    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aaec728-9b6b-4d36-af54-3e138d400a01" containerName="payload-processing"
Nov 02 23:57:06.977087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.976918    4294 cpu_manager.go:414] "containerMap: removing container" podUID="33dd7ffb-c342-476c-8ab6-6370fff64b9b" containerName="payload-processing"
Nov 02 23:57:06.977087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.976925    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33dd7ffb-c342-476c-8ab6-6370fff64b9b" containerName="payload-processing"
Nov 02 23:57:06.977087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.976946    4294 cpu_manager.go:414] "containerMap: removing container" podUID="e1728df9-e8b4-4c64-adc6-9178bdd638d8" containerName="payload-processing"
Nov 02 23:57:06.977087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.976952    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1728df9-e8b4-4c64-adc6-9178bdd638d8" containerName="payload-processing"
Nov 02 23:57:06.977087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.976964    4294 cpu_manager.go:414] "containerMap: removing container" podUID="d1b342a5-4d24-4147-921c-21cfa6b1f9ee" containerName="istio-proxy"
Nov 02 23:57:06.977087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.976971    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1b342a5-4d24-4147-921c-21cfa6b1f9ee" containerName="istio-proxy"
Nov 02 23:57:06.977087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.977002    4294 cpu_manager.go:414] "containerMap: removing container" podUID="25addf9e-2173-4971-9112-02e9857a65d4" containerName="payload-processing"
Nov 02 23:57:06.977087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.977010    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25addf9e-2173-4971-9112-02e9857a65d4" containerName="payload-processing"
Nov 02 23:57:06.977087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.977020    4294 cpu_manager.go:414] "containerMap: removing container" podUID="24256826-ee09-4791-af71-55d2e820dfdc" containerName="payload-pre-processing"
Nov 02 23:57:06.977087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.977027    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24256826-ee09-4791-af71-55d2e820dfdc" containerName="payload-pre-processing"
Nov 02 23:57:06.977087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.977054    4294 cpu_manager.go:414] "containerMap: removing container" podUID="d1b342a5-4d24-4147-921c-21cfa6b1f9ee" containerName="istio-proxy"
Nov 02 23:57:06.977087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.977062    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1b342a5-4d24-4147-921c-21cfa6b1f9ee" containerName="istio-proxy"
Nov 02 23:57:06.977887 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.977212    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1728df9-e8b4-4c64-adc6-9178bdd638d8" containerName="payload-processing"
Nov 02 23:57:06.977887 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.977228    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1b342a5-4d24-4147-921c-21cfa6b1f9ee" containerName="istio-proxy"
Nov 02 23:57:06.977887 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.977238    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="25addf9e-2173-4971-9112-02e9857a65d4" containerName="payload-processing"
Nov 02 23:57:06.977887 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.977248    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="24256826-ee09-4791-af71-55d2e820dfdc" containerName="payload-pre-processing"
Nov 02 23:57:06.977887 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.977261    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1b342a5-4d24-4147-921c-21cfa6b1f9ee" containerName="istio-proxy"
Nov 02 23:57:06.977887 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.977276    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d129273b-8713-4065-b518-7001a19aefd1" containerName="maas-api"
Nov 02 23:57:06.977887 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.977285    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfca5461-e8a7-4538-9e13-26aeca11ed47" containerName="payload-pre-processing"
Nov 02 23:57:06.977887 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.977295    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aaec728-9b6b-4d36-af54-3e138d400a01" containerName="payload-processing"
Nov 02 23:57:06.977887 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.977306    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="33dd7ffb-c342-476c-8ab6-6370fff64b9b" containerName="payload-processing"
Nov 02 23:57:06.982619 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:06.982598    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"
Nov 02 23:57:07.032832 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.032804    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 23:57:07.033047 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.033026    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-nqg67\""
Nov 02 23:57:07.033217 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.033196    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"]
Nov 02 23:57:07.077449 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.077422    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-eb3055757321\""
Nov 02 23:57:07.077449 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.077443    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 23:57:07.077620 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.077447    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 23:57:07.077620 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.077427    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 23:57:07.099136 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.099099    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nldkw\" (UniqueName: \"kubernetes.io/projected/5817e183-c338-485c-b383-9260c19dd283-kube-api-access-nldkw\") pod \"payload-processing-external-model-6ff598f8d6-89r5j\" (UID: \"5817e183-c338-485c-b383-9260c19dd283\") " pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"
Nov 02 23:57:07.099334 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.099308    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5817e183-c338-485c-b383-9260c19dd283-provider-credentials\") pod \"payload-processing-external-model-6ff598f8d6-89r5j\" (UID: \"5817e183-c338-485c-b383-9260c19dd283\") " pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"
Nov 02 23:57:07.099422 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.099404    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5817e183-c338-485c-b383-9260c19dd283-routing-overlay-volume\") pod \"payload-processing-external-model-6ff598f8d6-89r5j\" (UID: \"5817e183-c338-485c-b383-9260c19dd283\") " pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"
Nov 02 23:57:07.099496 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.099481    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5817e183-c338-485c-b383-9260c19dd283-plugins-config-volume\") pod \"payload-processing-external-model-6ff598f8d6-89r5j\" (UID: \"5817e183-c338-485c-b383-9260c19dd283\") " pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"
Nov 02 23:57:07.200943 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.200854    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5817e183-c338-485c-b383-9260c19dd283-provider-credentials\") pod \"payload-processing-external-model-6ff598f8d6-89r5j\" (UID: \"5817e183-c338-485c-b383-9260c19dd283\") " pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"
Nov 02 23:57:07.200943 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.200919    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5817e183-c338-485c-b383-9260c19dd283-routing-overlay-volume\") pod \"payload-processing-external-model-6ff598f8d6-89r5j\" (UID: \"5817e183-c338-485c-b383-9260c19dd283\") " pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"
Nov 02 23:57:07.201225 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.200964    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5817e183-c338-485c-b383-9260c19dd283-plugins-config-volume\") pod \"payload-processing-external-model-6ff598f8d6-89r5j\" (UID: \"5817e183-c338-485c-b383-9260c19dd283\") " pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"
Nov 02 23:57:07.201225 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.201038    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nldkw\" (UniqueName: \"kubernetes.io/projected/5817e183-c338-485c-b383-9260c19dd283-kube-api-access-nldkw\") pod \"payload-processing-external-model-6ff598f8d6-89r5j\" (UID: \"5817e183-c338-485c-b383-9260c19dd283\") " pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"
Nov 02 23:57:07.201624 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.201597    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5817e183-c338-485c-b383-9260c19dd283-routing-overlay-volume\") pod \"payload-processing-external-model-6ff598f8d6-89r5j\" (UID: \"5817e183-c338-485c-b383-9260c19dd283\") " pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"
Nov 02 23:57:07.201723 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.201683    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5817e183-c338-485c-b383-9260c19dd283-plugins-config-volume\") pod \"payload-processing-external-model-6ff598f8d6-89r5j\" (UID: \"5817e183-c338-485c-b383-9260c19dd283\") " pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"
Nov 02 23:57:07.203502 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.203484    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5817e183-c338-485c-b383-9260c19dd283-provider-credentials\") pod \"payload-processing-external-model-6ff598f8d6-89r5j\" (UID: \"5817e183-c338-485c-b383-9260c19dd283\") " pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"
Nov 02 23:57:07.255558 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.255217    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nldkw\" (UniqueName: \"kubernetes.io/projected/5817e183-c338-485c-b383-9260c19dd283-kube-api-access-nldkw\") pod \"payload-processing-external-model-6ff598f8d6-89r5j\" (UID: \"5817e183-c338-485c-b383-9260c19dd283\") " pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"
Nov 02 23:57:07.295874 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.295834    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"
Nov 02 23:57:07.460547 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.460464    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:07.675197 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.675161    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:07.693348 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.693323    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"]
Nov 02 23:57:07.694322 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:57:07.694298    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5817e183_c338_485c_b383_9260c19dd283.slice/crio-2365b97dab6d204d164ac81e43b49de7616488eb833957f1c444ea5fdcbde8ae WatchSource:0}: Error finding container 2365b97dab6d204d164ac81e43b49de7616488eb833957f1c444ea5fdcbde8ae: Status 404 returned error can't find the container with id 2365b97dab6d204d164ac81e43b49de7616488eb833957f1c444ea5fdcbde8ae
Nov 02 23:57:07.828313 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:07.828262    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:08.135552 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:08.135508    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j" event={"ID":"5817e183-c338-485c-b383-9260c19dd283","Type":"ContainerStarted","Data":"06d820edfba217e800939961d9f3db84058fd7f10a25fd8804f6d8c9018a0034"}
Nov 02 23:57:08.135739 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:08.135607    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"
Nov 02 23:57:08.135739 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:08.135630    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j" event={"ID":"5817e183-c338-485c-b383-9260c19dd283","Type":"ContainerStarted","Data":"2365b97dab6d204d164ac81e43b49de7616488eb833957f1c444ea5fdcbde8ae"}
Nov 02 23:57:08.266323 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:08.266254    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j" podStartSLOduration=2.266237133 podStartE2EDuration="2.266237133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:57:08.263241317 +0000 UTC m=+3867.831596104" watchObservedRunningTime="2026-11-02 23:57:08.266237133 +0000 UTC m=+3867.834591920"
Nov 02 23:57:09.160677 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:09.160634    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:09.333913 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:09.333885    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:09.608414 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:09.608381    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:09.789034 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:09.789001    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:11.610476 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:11.610438    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:11.751134 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:57:11.751100    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:57:11.767911 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:11.767883    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:12.514964 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:12.514924    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:12.683253 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:12.683213    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:13.099020 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:13.098972    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:13.569281 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:13.569250    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:14.432375 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:14.432340    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:17.309832 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:17.309797    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:17.529126 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:17.529092    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:19.145106 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:19.145072    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"
Nov 02 23:57:19.170051 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:19.169964    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:20.175416 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:20.175378    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:21.330548 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:21.330516    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-64887bb858-xh446"]
Nov 02 23:57:21.334935 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:21.334911    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446"
Nov 02 23:57:21.379297 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:21.379259    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials\") pod \"payload-processing-external-model-64887bb858-xh446\" (UID: \"6ae49deb-156f-4f33-a81d-06bc93b7dcb8\") " pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446"
Nov 02 23:57:21.379468 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:21.379384    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4rdq\" (UniqueName: \"kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-kube-api-access-z4rdq\") pod \"payload-processing-external-model-64887bb858-xh446\" (UID: \"6ae49deb-156f-4f33-a81d-06bc93b7dcb8\") " pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446"
Nov 02 23:57:21.379516 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:21.379482    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-plugins-config-volume\") pod \"payload-processing-external-model-64887bb858-xh446\" (UID: \"6ae49deb-156f-4f33-a81d-06bc93b7dcb8\") " pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446"
Nov 02 23:57:21.379573 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:21.379549    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-routing-overlay-volume\") pod \"payload-processing-external-model-64887bb858-xh446\" (UID: \"6ae49deb-156f-4f33-a81d-06bc93b7dcb8\") " pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446"
Nov 02 23:57:21.385039 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:21.385015    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a610cb5fdd81\""
Nov 02 23:57:21.385039 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:21.385024    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7ea550d38410\""
Nov 02 23:57:21.435805 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:21.435773    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-64887bb858-xh446"]
Nov 02 23:57:21.480680 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:21.480641    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4rdq\" (UniqueName: \"kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-kube-api-access-z4rdq\") pod \"payload-processing-external-model-64887bb858-xh446\" (UID: \"6ae49deb-156f-4f33-a81d-06bc93b7dcb8\") " pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446"
Nov 02 23:57:21.480885 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:21.480735    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-plugins-config-volume\") pod \"payload-processing-external-model-64887bb858-xh446\" (UID: \"6ae49deb-156f-4f33-a81d-06bc93b7dcb8\") " pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446"
Nov 02 23:57:21.480885 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:21.480759    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-routing-overlay-volume\") pod \"payload-processing-external-model-64887bb858-xh446\" (UID: \"6ae49deb-156f-4f33-a81d-06bc93b7dcb8\") " pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446"
Nov 02 23:57:21.480885 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:21.480851    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials\") pod \"payload-processing-external-model-64887bb858-xh446\" (UID: \"6ae49deb-156f-4f33-a81d-06bc93b7dcb8\") " pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446"
Nov 02 23:57:21.481667 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:21.481639    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-plugins-config-volume\") pod \"payload-processing-external-model-64887bb858-xh446\" (UID: \"6ae49deb-156f-4f33-a81d-06bc93b7dcb8\") " pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446"
Nov 02 23:57:21.481785 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:21.481648    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-routing-overlay-volume\") pod \"payload-processing-external-model-64887bb858-xh446\" (UID: \"6ae49deb-156f-4f33-a81d-06bc93b7dcb8\") " pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446"
Nov 02 23:57:21.483691 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:21.483661    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials\") pod \"payload-processing-external-model-64887bb858-xh446\" (UID: \"6ae49deb-156f-4f33-a81d-06bc93b7dcb8\") " pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446"
Nov 02 23:57:21.535607 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:21.535570    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4rdq\" (UniqueName: \"kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-kube-api-access-z4rdq\") pod \"payload-processing-external-model-64887bb858-xh446\" (UID: \"6ae49deb-156f-4f33-a81d-06bc93b7dcb8\") " pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446"
Nov 02 23:57:21.649565 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:21.649486    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446"
Nov 02 23:57:22.017011 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:57:22.016968    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae49deb_156f_4f33_a81d_06bc93b7dcb8.slice/crio-7ba9cb4a44a600905ee143ac134a35a645241f3ded3f90ea9a02b35e031a8b3c WatchSource:0}: Error finding container 7ba9cb4a44a600905ee143ac134a35a645241f3ded3f90ea9a02b35e031a8b3c: Status 404 returned error can't find the container with id 7ba9cb4a44a600905ee143ac134a35a645241f3ded3f90ea9a02b35e031a8b3c
Nov 02 23:57:22.017392 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:22.017366    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-64887bb858-xh446"]
Nov 02 23:57:22.226885 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:22.226844    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446" event={"ID":"6ae49deb-156f-4f33-a81d-06bc93b7dcb8","Type":"ContainerStarted","Data":"9141eb06037c5b06f23328268a22d117fa164ed604503e0485e50ed0cacb49a8"}
Nov 02 23:57:22.227127 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:22.226916    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446"
Nov 02 23:57:22.227127 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:22.226935    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446" event={"ID":"6ae49deb-156f-4f33-a81d-06bc93b7dcb8","Type":"ContainerStarted","Data":"7ba9cb4a44a600905ee143ac134a35a645241f3ded3f90ea9a02b35e031a8b3c"}
Nov 02 23:57:22.337821 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:22.337711    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446" podStartSLOduration=1.337696066 podStartE2EDuration="1.337696066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:57:22.335819161 +0000 UTC m=+3881.904173948" watchObservedRunningTime="2026-11-02 23:57:22.337696066 +0000 UTC m=+3881.906050853"
Nov 02 23:57:23.658958 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:23.658924    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:28.246332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:28.246297    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:32.831683 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:32.831632    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:33.236170 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:33.236087    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446"
Nov 02 23:57:33.401761 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:33.401726    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"]
Nov 02 23:57:37.386457 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:37.386424    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:38.402769 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:38.402722    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j" podUID="5817e183-c338-485c-b383-9260c19dd283" containerName="payload-processing" containerID="cri-o://06d820edfba217e800939961d9f3db84058fd7f10a25fd8804f6d8c9018a0034" gracePeriod=25
Nov 02 23:57:38.856284 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:38.855883    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"
Nov 02 23:57:39.036120 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.036016    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5817e183-c338-485c-b383-9260c19dd283-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5817e183-c338-485c-b383-9260c19dd283-routing-overlay-volume\") pod \"5817e183-c338-485c-b383-9260c19dd283\" (UID: \"5817e183-c338-485c-b383-9260c19dd283\") "
Nov 02 23:57:39.036492 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.036458    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5817e183-c338-485c-b383-9260c19dd283-kube-api-access-nldkw\" (UniqueName: \"kubernetes.io/projected/5817e183-c338-485c-b383-9260c19dd283-kube-api-access-nldkw\") pod \"5817e183-c338-485c-b383-9260c19dd283\" (UID: \"5817e183-c338-485c-b383-9260c19dd283\") "
Nov 02 23:57:39.036695 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.036667    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5817e183-c338-485c-b383-9260c19dd283-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5817e183-c338-485c-b383-9260c19dd283-plugins-config-volume\") pod \"5817e183-c338-485c-b383-9260c19dd283\" (UID: \"5817e183-c338-485c-b383-9260c19dd283\") "
Nov 02 23:57:39.036845 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.036829    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5817e183-c338-485c-b383-9260c19dd283-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5817e183-c338-485c-b383-9260c19dd283-provider-credentials\") pod \"5817e183-c338-485c-b383-9260c19dd283\" (UID: \"5817e183-c338-485c-b383-9260c19dd283\") "
Nov 02 23:57:39.037503 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.037480    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5817e183-c338-485c-b383-9260c19dd283-plugins-config-volume" pod "5817e183-c338-485c-b383-9260c19dd283" (UID: "5817e183-c338-485c-b383-9260c19dd283"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:57:39.039072 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.037752    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5817e183-c338-485c-b383-9260c19dd283-routing-overlay-volume" pod "5817e183-c338-485c-b383-9260c19dd283" (UID: "5817e183-c338-485c-b383-9260c19dd283"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:57:39.039489 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.039331    4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5817e183-c338-485c-b383-9260c19dd283-routing-overlay-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:39.039489 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.039365    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5817e183-c338-485c-b383-9260c19dd283-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:39.039758 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.039580    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5817e183-c338-485c-b383-9260c19dd283-kube-api-access-nldkw" pod "5817e183-c338-485c-b383-9260c19dd283" (UID: "5817e183-c338-485c-b383-9260c19dd283"). InnerVolumeSpecName "kube-api-access-nldkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:57:39.042778 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.042715    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5817e183-c338-485c-b383-9260c19dd283-provider-credentials" pod "5817e183-c338-485c-b383-9260c19dd283" (UID: "5817e183-c338-485c-b383-9260c19dd283"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:57:39.141092 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.141055    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nldkw\" (UniqueName: \"kubernetes.io/projected/5817e183-c338-485c-b383-9260c19dd283-kube-api-access-nldkw\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:39.141092 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.141088    4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5817e183-c338-485c-b383-9260c19dd283-provider-credentials\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:39.346815 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.346718    4294 generic.go:356] "Generic (PLEG): container finished" podID="5817e183-c338-485c-b383-9260c19dd283" containerID="06d820edfba217e800939961d9f3db84058fd7f10a25fd8804f6d8c9018a0034" exitCode=0
Nov 02 23:57:39.346815 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.346774    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"
Nov 02 23:57:39.346815 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.346787    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j" event={"ID":"5817e183-c338-485c-b383-9260c19dd283","Type":"ContainerDied","Data":"06d820edfba217e800939961d9f3db84058fd7f10a25fd8804f6d8c9018a0034"}
Nov 02 23:57:39.347085 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.346871    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j" event={"ID":"5817e183-c338-485c-b383-9260c19dd283","Type":"ContainerDied","Data":"2365b97dab6d204d164ac81e43b49de7616488eb833957f1c444ea5fdcbde8ae"}
Nov 02 23:57:39.347085 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.346901    4294 scope.go:122] "RemoveContainer" containerID="06d820edfba217e800939961d9f3db84058fd7f10a25fd8804f6d8c9018a0034"
Nov 02 23:57:39.357262 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.357239    4294 scope.go:122] "RemoveContainer" containerID="06d820edfba217e800939961d9f3db84058fd7f10a25fd8804f6d8c9018a0034"
Nov 02 23:57:39.357606 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:57:39.357585    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d820edfba217e800939961d9f3db84058fd7f10a25fd8804f6d8c9018a0034\": container with ID starting with 06d820edfba217e800939961d9f3db84058fd7f10a25fd8804f6d8c9018a0034 not found: ID does not exist" containerID="06d820edfba217e800939961d9f3db84058fd7f10a25fd8804f6d8c9018a0034"
Nov 02 23:57:39.357685 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.357616    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d820edfba217e800939961d9f3db84058fd7f10a25fd8804f6d8c9018a0034"} err="failed to get container status \"06d820edfba217e800939961d9f3db84058fd7f10a25fd8804f6d8c9018a0034\": rpc error: code = NotFound desc = could not find container \"06d820edfba217e800939961d9f3db84058fd7f10a25fd8804f6d8c9018a0034\": container with ID starting with 06d820edfba217e800939961d9f3db84058fd7f10a25fd8804f6d8c9018a0034 not found: ID does not exist"
Nov 02 23:57:39.508357 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.508315    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"]
Nov 02 23:57:39.557315 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:39.557273    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ff598f8d6-89r5j"]
Nov 02 23:57:40.550081 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:40.550050    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5817e183-c338-485c-b383-9260c19dd283" path="/var/lib/kubelet/pods/5817e183-c338-485c-b383-9260c19dd283/volumes"
Nov 02 23:57:41.183050 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:41.183011    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"]
Nov 02 23:57:41.183273 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:41.183245    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j" podUID="88859309-dc51-43ee-a461-ec486b3f268f" containerName="maas-api" containerID="cri-o://a2c03b14127f1cf7647e5b78e6b2092e93378b80f0643d0ac035c005aefd12b3" gracePeriod=30
Nov 02 23:57:41.305772 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:41.305737    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:41.367783 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:41.367750    4294 generic.go:356] "Generic (PLEG): container finished" podID="88859309-dc51-43ee-a461-ec486b3f268f" containerID="a2c03b14127f1cf7647e5b78e6b2092e93378b80f0643d0ac035c005aefd12b3" exitCode=0
Nov 02 23:57:41.367954 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:41.367924    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j" event={"ID":"88859309-dc51-43ee-a461-ec486b3f268f","Type":"ContainerDied","Data":"a2c03b14127f1cf7647e5b78e6b2092e93378b80f0643d0ac035c005aefd12b3"}
Nov 02 23:57:41.447436 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:41.447367    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"
Nov 02 23:57:41.471026 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:41.470989    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88859309-dc51-43ee-a461-ec486b3f268f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/88859309-dc51-43ee-a461-ec486b3f268f-metrics-certs\") pod \"88859309-dc51-43ee-a461-ec486b3f268f\" (UID: \"88859309-dc51-43ee-a461-ec486b3f268f\") "
Nov 02 23:57:41.471215 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:41.471120    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88859309-dc51-43ee-a461-ec486b3f268f-kube-api-access-z82w5\" (UniqueName: \"kubernetes.io/projected/88859309-dc51-43ee-a461-ec486b3f268f-kube-api-access-z82w5\") pod \"88859309-dc51-43ee-a461-ec486b3f268f\" (UID: \"88859309-dc51-43ee-a461-ec486b3f268f\") "
Nov 02 23:57:41.471215 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:41.471143    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88859309-dc51-43ee-a461-ec486b3f268f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88859309-dc51-43ee-a461-ec486b3f268f-openshift-service-ca\") pod \"88859309-dc51-43ee-a461-ec486b3f268f\" (UID: \"88859309-dc51-43ee-a461-ec486b3f268f\") "
Nov 02 23:57:41.471331 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:41.471247    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88859309-dc51-43ee-a461-ec486b3f268f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88859309-dc51-43ee-a461-ec486b3f268f-maas-api-tls\") pod \"88859309-dc51-43ee-a461-ec486b3f268f\" (UID: \"88859309-dc51-43ee-a461-ec486b3f268f\") "
Nov 02 23:57:41.471596 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:41.471571    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88859309-dc51-43ee-a461-ec486b3f268f-openshift-service-ca" pod "88859309-dc51-43ee-a461-ec486b3f268f" (UID: "88859309-dc51-43ee-a461-ec486b3f268f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:57:41.473251 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:41.473221    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88859309-dc51-43ee-a461-ec486b3f268f-metrics-certs" pod "88859309-dc51-43ee-a461-ec486b3f268f" (UID: "88859309-dc51-43ee-a461-ec486b3f268f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:57:41.473344 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:41.473292    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88859309-dc51-43ee-a461-ec486b3f268f-maas-api-tls" pod "88859309-dc51-43ee-a461-ec486b3f268f" (UID: "88859309-dc51-43ee-a461-ec486b3f268f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:57:41.473443 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:41.473423    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88859309-dc51-43ee-a461-ec486b3f268f-kube-api-access-z82w5" pod "88859309-dc51-43ee-a461-ec486b3f268f" (UID: "88859309-dc51-43ee-a461-ec486b3f268f"). InnerVolumeSpecName "kube-api-access-z82w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:57:41.573639 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:41.573595    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88859309-dc51-43ee-a461-ec486b3f268f-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:41.573639 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:41.573628    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88859309-dc51-43ee-a461-ec486b3f268f-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:41.573639 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:41.573643    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z82w5\" (UniqueName: \"kubernetes.io/projected/88859309-dc51-43ee-a461-ec486b3f268f-kube-api-access-z82w5\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:41.574307 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:41.573654    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88859309-dc51-43ee-a461-ec486b3f268f-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:41.650724 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:41.650693    4294 scope.go:122] "RemoveContainer" containerID="e3613be192d9a2c044811b702cd17cc07237015dd7a7e2aa81a0d42f263cd717"
Nov 02 23:57:41.751244 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:57:41.751200    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:57:42.375649 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:42.375622    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"
Nov 02 23:57:42.375649 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:42.375634    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j" event={"ID":"88859309-dc51-43ee-a461-ec486b3f268f","Type":"ContainerDied","Data":"4d9057e47042737454154b3542c6c17d2238c6e2585720ff84aeb47aa57d9337"}
Nov 02 23:57:42.375933 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:42.375737    4294 scope.go:122] "RemoveContainer" containerID="a2c03b14127f1cf7647e5b78e6b2092e93378b80f0643d0ac035c005aefd12b3"
Nov 02 23:57:42.533628 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:42.533593    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"]
Nov 02 23:57:42.580687 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:42.580654    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c11b50-8696796f47-5fw9j"]
Nov 02 23:57:43.090679 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:43.090640    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:43.321031 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:43.320956    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"]
Nov 02 23:57:43.321271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:43.321245    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7" podUID="248cacc6-b671-459f-b879-6ce8ddbc72c3" containerName="istio-proxy" containerID="cri-o://a38dbaf41aa9fcd103b3353bfa27b5d4a359acde4dad86a68296edf7ebe803d9" gracePeriod=30
Nov 02 23:57:43.393785 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:43.393703    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:43.560792 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:43.560756    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:44.549468 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:44.549433    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88859309-dc51-43ee-a461-ec486b3f268f" path="/var/lib/kubelet/pods/88859309-dc51-43ee-a461-ec486b3f268f/volumes"
Nov 02 23:57:45.123004 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:45.122936    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:45.714521 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:45.714475    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:45.980506 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:45.980422    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:46.316592 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:46.316553    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:46.513485 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:46.513448    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:47.063070 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:47.063031    4294 patch_prober.go:29] interesting pod/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 23:57:47.063479 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:47.063106    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7" podUID="248cacc6-b671-459f-b879-6ce8ddbc72c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 23:57:47.280332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:47.280299    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:48.596042 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.595961    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:57:48.662330 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.662285    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-credential-socket\") pod \"248cacc6-b671-459f-b879-6ce8ddbc72c3\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") "
Nov 02 23:57:48.662330 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.662344    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248cacc6-b671-459f-b879-6ce8ddbc72c3-kube-api-access-pm4b4\" (UniqueName: \"kubernetes.io/projected/248cacc6-b671-459f-b879-6ce8ddbc72c3-kube-api-access-pm4b4\") pod \"248cacc6-b671-459f-b879-6ce8ddbc72c3\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") "
Nov 02 23:57:48.662610 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.662404    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-workload-certs\") pod \"248cacc6-b671-459f-b879-6ce8ddbc72c3\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") "
Nov 02 23:57:48.662610 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.662445    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-podinfo\") pod \"248cacc6-b671-459f-b879-6ce8ddbc72c3\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") "
Nov 02 23:57:48.662610 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.662470    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-token\" (UniqueName: \"kubernetes.io/projected/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-token\") pod \"248cacc6-b671-459f-b879-6ce8ddbc72c3\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") "
Nov 02 23:57:48.662610 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.662493    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/248cacc6-b671-459f-b879-6ce8ddbc72c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/248cacc6-b671-459f-b879-6ce8ddbc72c3-istiod-ca-cert\") pod \"248cacc6-b671-459f-b879-6ce8ddbc72c3\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") "
Nov 02 23:57:48.662610 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.662546    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-data\") pod \"248cacc6-b671-459f-b879-6ce8ddbc72c3\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") "
Nov 02 23:57:48.662886 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.662605    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-credential-socket" pod "248cacc6-b671-459f-b879-6ce8ddbc72c3" (UID: "248cacc6-b671-459f-b879-6ce8ddbc72c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:57:48.662886 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.662616    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-workload-socket\") pod \"248cacc6-b671-459f-b879-6ce8ddbc72c3\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") "
Nov 02 23:57:48.662886 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.662748    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-envoy\") pod \"248cacc6-b671-459f-b879-6ce8ddbc72c3\" (UID: \"248cacc6-b671-459f-b879-6ce8ddbc72c3\") "
Nov 02 23:57:48.662886 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.662767    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-workload-certs" pod "248cacc6-b671-459f-b879-6ce8ddbc72c3" (UID: "248cacc6-b671-459f-b879-6ce8ddbc72c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:57:48.663156 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.662961    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-workload-socket" pod "248cacc6-b671-459f-b879-6ce8ddbc72c3" (UID: "248cacc6-b671-459f-b879-6ce8ddbc72c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:57:48.663156 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.662976    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-data" pod "248cacc6-b671-459f-b879-6ce8ddbc72c3" (UID: "248cacc6-b671-459f-b879-6ce8ddbc72c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:57:48.663156 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.663091    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248cacc6-b671-459f-b879-6ce8ddbc72c3-istiod-ca-cert" pod "248cacc6-b671-459f-b879-6ce8ddbc72c3" (UID: "248cacc6-b671-459f-b879-6ce8ddbc72c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:57:48.663597 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.663573    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:48.663597 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.663598    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/248cacc6-b671-459f-b879-6ce8ddbc72c3-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:48.663806 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.663611    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:48.663806 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.663623    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:48.663806 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.663637    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:48.665090 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.665066    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-token" pod "248cacc6-b671-459f-b879-6ce8ddbc72c3" (UID: "248cacc6-b671-459f-b879-6ce8ddbc72c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:57:48.665309 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.665288    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248cacc6-b671-459f-b879-6ce8ddbc72c3-kube-api-access-pm4b4" pod "248cacc6-b671-459f-b879-6ce8ddbc72c3" (UID: "248cacc6-b671-459f-b879-6ce8ddbc72c3"). InnerVolumeSpecName "kube-api-access-pm4b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:57:48.665441 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.665417    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-podinfo" pod "248cacc6-b671-459f-b879-6ce8ddbc72c3" (UID: "248cacc6-b671-459f-b879-6ce8ddbc72c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 23:57:48.665615 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.665598    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-envoy" pod "248cacc6-b671-459f-b879-6ce8ddbc72c3" (UID: "248cacc6-b671-459f-b879-6ce8ddbc72c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 23:57:48.764397 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.764361    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:48.764397 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.764390    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:48.764397 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.764399    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/248cacc6-b671-459f-b879-6ce8ddbc72c3-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:48.764397 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:48.764408    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm4b4\" (UniqueName: \"kubernetes.io/projected/248cacc6-b671-459f-b879-6ce8ddbc72c3-kube-api-access-pm4b4\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:57:49.223023 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:49.222958    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:49.425898 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:49.425867    4294 generic.go:356] "Generic (PLEG): container finished" podID="248cacc6-b671-459f-b879-6ce8ddbc72c3" containerID="a38dbaf41aa9fcd103b3353bfa27b5d4a359acde4dad86a68296edf7ebe803d9" exitCode=0
Nov 02 23:57:49.426118 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:49.425918    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"
Nov 02 23:57:49.426118 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:49.425954    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7" event={"ID":"248cacc6-b671-459f-b879-6ce8ddbc72c3","Type":"ContainerDied","Data":"a38dbaf41aa9fcd103b3353bfa27b5d4a359acde4dad86a68296edf7ebe803d9"}
Nov 02 23:57:49.426118 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:49.426030    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7" event={"ID":"248cacc6-b671-459f-b879-6ce8ddbc72c3","Type":"ContainerDied","Data":"fe7a88b04135c29fe7577ef4d05b0205e4b5666021150e85e351b516ae726269"}
Nov 02 23:57:49.426118 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:49.426055    4294 scope.go:122] "RemoveContainer" containerID="a38dbaf41aa9fcd103b3353bfa27b5d4a359acde4dad86a68296edf7ebe803d9"
Nov 02 23:57:49.437114 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:49.437092    4294 scope.go:122] "RemoveContainer" containerID="dffe15488bc2fe12c16140db8f75d0dc19402e1913d9050fc3349891b63b615b"
Nov 02 23:57:49.447151 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:49.446931    4294 scope.go:122] "RemoveContainer" containerID="a38dbaf41aa9fcd103b3353bfa27b5d4a359acde4dad86a68296edf7ebe803d9"
Nov 02 23:57:49.447289 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:57:49.447272    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a38dbaf41aa9fcd103b3353bfa27b5d4a359acde4dad86a68296edf7ebe803d9\": container with ID starting with a38dbaf41aa9fcd103b3353bfa27b5d4a359acde4dad86a68296edf7ebe803d9 not found: ID does not exist" containerID="a38dbaf41aa9fcd103b3353bfa27b5d4a359acde4dad86a68296edf7ebe803d9"
Nov 02 23:57:49.447332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:49.447299    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a38dbaf41aa9fcd103b3353bfa27b5d4a359acde4dad86a68296edf7ebe803d9"} err="failed to get container status \"a38dbaf41aa9fcd103b3353bfa27b5d4a359acde4dad86a68296edf7ebe803d9\": rpc error: code = NotFound desc = could not find container \"a38dbaf41aa9fcd103b3353bfa27b5d4a359acde4dad86a68296edf7ebe803d9\": container with ID starting with a38dbaf41aa9fcd103b3353bfa27b5d4a359acde4dad86a68296edf7ebe803d9 not found: ID does not exist"
Nov 02 23:57:49.447332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:49.447321    4294 scope.go:122] "RemoveContainer" containerID="dffe15488bc2fe12c16140db8f75d0dc19402e1913d9050fc3349891b63b615b"
Nov 02 23:57:49.447546 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:57:49.447530    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dffe15488bc2fe12c16140db8f75d0dc19402e1913d9050fc3349891b63b615b\": container with ID starting with dffe15488bc2fe12c16140db8f75d0dc19402e1913d9050fc3349891b63b615b not found: ID does not exist" containerID="dffe15488bc2fe12c16140db8f75d0dc19402e1913d9050fc3349891b63b615b"
Nov 02 23:57:49.447589 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:49.447552    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dffe15488bc2fe12c16140db8f75d0dc19402e1913d9050fc3349891b63b615b"} err="failed to get container status \"dffe15488bc2fe12c16140db8f75d0dc19402e1913d9050fc3349891b63b615b\": rpc error: code = NotFound desc = could not find container \"dffe15488bc2fe12c16140db8f75d0dc19402e1913d9050fc3349891b63b615b\": container with ID starting with dffe15488bc2fe12c16140db8f75d0dc19402e1913d9050fc3349891b63b615b not found: ID does not exist"
Nov 02 23:57:49.581499 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:49.581466    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:49.584335 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:49.584311    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"]
Nov 02 23:57:49.631287 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:49.631253    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-c11b50-openshift-default-7448fd5ff-p2cl7"]
Nov 02 23:57:49.754104 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:49.754069    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:49.912268 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:49.912182    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:50.548167 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:50.548133    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="248cacc6-b671-459f-b879-6ce8ddbc72c3" path="/var/lib/kubelet/pods/248cacc6-b671-459f-b879-6ce8ddbc72c3/volumes"
Nov 02 23:57:51.633975 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:51.633936    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:51.793221 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:51.793188    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:52.684413 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:52.684381    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:52.846484 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:52.846445    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:53.063166 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.063125    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9"]
Nov 02 23:57:53.063748 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.063734    4294 cpu_manager.go:414] "containerMap: removing container" podUID="5817e183-c338-485c-b383-9260c19dd283" containerName="payload-processing"
Nov 02 23:57:53.063801 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.063750    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5817e183-c338-485c-b383-9260c19dd283" containerName="payload-processing"
Nov 02 23:57:53.063801 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.063767    4294 cpu_manager.go:414] "containerMap: removing container" podUID="248cacc6-b671-459f-b879-6ce8ddbc72c3" containerName="istio-proxy"
Nov 02 23:57:53.063801 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.063772    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248cacc6-b671-459f-b879-6ce8ddbc72c3" containerName="istio-proxy"
Nov 02 23:57:53.063801 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.063783    4294 cpu_manager.go:414] "containerMap: removing container" podUID="248cacc6-b671-459f-b879-6ce8ddbc72c3" containerName="istio-proxy"
Nov 02 23:57:53.063801 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.063789    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248cacc6-b671-459f-b879-6ce8ddbc72c3" containerName="istio-proxy"
Nov 02 23:57:53.063958 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.063814    4294 cpu_manager.go:414] "containerMap: removing container" podUID="88859309-dc51-43ee-a461-ec486b3f268f" containerName="maas-api"
Nov 02 23:57:53.063958 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.063818    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88859309-dc51-43ee-a461-ec486b3f268f" containerName="maas-api"
Nov 02 23:57:53.063958 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.063898    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="88859309-dc51-43ee-a461-ec486b3f268f" containerName="maas-api"
Nov 02 23:57:53.063958 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.063910    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="5817e183-c338-485c-b383-9260c19dd283" containerName="payload-processing"
Nov 02 23:57:53.063958 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.063917    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="248cacc6-b671-459f-b879-6ce8ddbc72c3" containerName="istio-proxy"
Nov 02 23:57:53.063958 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.063922    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="248cacc6-b671-459f-b879-6ce8ddbc72c3" containerName="istio-proxy"
Nov 02 23:57:53.068840 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.068823    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9"
Nov 02 23:57:53.120924 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.120888    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9"]
Nov 02 23:57:53.164225 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.164187    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-ad18ae\""
Nov 02 23:57:53.164225 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.164193    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-ad18ae-dockercfg-9tnk6\""
Nov 02 23:57:53.225394 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.225360    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msrbm\" (UniqueName: \"kubernetes.io/projected/0bf1c293-845f-4179-a74e-8203f5d07479-kube-api-access-msrbm\") pod \"payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9\" (UID: \"0bf1c293-845f-4179-a74e-8203f5d07479\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9"
Nov 02 23:57:53.225585 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.225541    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf1c293-845f-4179-a74e-8203f5d07479-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9\" (UID: \"0bf1c293-845f-4179-a74e-8203f5d07479\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9"
Nov 02 23:57:53.227237 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.227218    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2"]
Nov 02 23:57:53.231332 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.231317    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2"
Nov 02 23:57:53.283018 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.282988    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2"]
Nov 02 23:57:53.326788 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.326710    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxc89\" (UniqueName: \"kubernetes.io/projected/37c6298e-b0e8-4f3f-80d7-62a04699cbc5-kube-api-access-gxc89\") pod \"payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2\" (UID: \"37c6298e-b0e8-4f3f-80d7-62a04699cbc5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2"
Nov 02 23:57:53.326788 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.326770    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf1c293-845f-4179-a74e-8203f5d07479-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9\" (UID: \"0bf1c293-845f-4179-a74e-8203f5d07479\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9"
Nov 02 23:57:53.327019 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.326879    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37c6298e-b0e8-4f3f-80d7-62a04699cbc5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2\" (UID: \"37c6298e-b0e8-4f3f-80d7-62a04699cbc5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2"
Nov 02 23:57:53.327087 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.327063    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msrbm\" (UniqueName: \"kubernetes.io/projected/0bf1c293-845f-4179-a74e-8203f5d07479-kube-api-access-msrbm\") pod \"payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9\" (UID: \"0bf1c293-845f-4179-a74e-8203f5d07479\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9"
Nov 02 23:57:53.327652 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.327634    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf1c293-845f-4179-a74e-8203f5d07479-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9\" (UID: \"0bf1c293-845f-4179-a74e-8203f5d07479\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9"
Nov 02 23:57:53.381808 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.381773    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msrbm\" (UniqueName: \"kubernetes.io/projected/0bf1c293-845f-4179-a74e-8203f5d07479-kube-api-access-msrbm\") pod \"payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9\" (UID: \"0bf1c293-845f-4179-a74e-8203f5d07479\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9"
Nov 02 23:57:53.382419 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.382393    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9"
Nov 02 23:57:53.428714 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.428674    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxc89\" (UniqueName: \"kubernetes.io/projected/37c6298e-b0e8-4f3f-80d7-62a04699cbc5-kube-api-access-gxc89\") pod \"payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2\" (UID: \"37c6298e-b0e8-4f3f-80d7-62a04699cbc5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2"
Nov 02 23:57:53.428925 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.428751    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37c6298e-b0e8-4f3f-80d7-62a04699cbc5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2\" (UID: \"37c6298e-b0e8-4f3f-80d7-62a04699cbc5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2"
Nov 02 23:57:53.429466 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.429443    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37c6298e-b0e8-4f3f-80d7-62a04699cbc5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2\" (UID: \"37c6298e-b0e8-4f3f-80d7-62a04699cbc5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2"
Nov 02 23:57:53.484528 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.484491    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxc89\" (UniqueName: \"kubernetes.io/projected/37c6298e-b0e8-4f3f-80d7-62a04699cbc5-kube-api-access-gxc89\") pod \"payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2\" (UID: \"37c6298e-b0e8-4f3f-80d7-62a04699cbc5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2"
Nov 02 23:57:53.545538 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.545505    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2"
Nov 02 23:57:53.751770 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:57:53.751732    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf1c293_845f_4179_a74e_8203f5d07479.slice/crio-8a31b5229b6702faf4b65acd27acd6ad09aa4b3af5055af479f157cf5c22ada3 WatchSource:0}: Error finding container 8a31b5229b6702faf4b65acd27acd6ad09aa4b3af5055af479f157cf5c22ada3: Status 404 returned error can't find the container with id 8a31b5229b6702faf4b65acd27acd6ad09aa4b3af5055af479f157cf5c22ada3
Nov 02 23:57:53.753044 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.753010    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9"]
Nov 02 23:57:53.769551 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.769521    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:53.915564 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:53.915532    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2"]
Nov 02 23:57:53.915719 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:57:53.915632    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c6298e_b0e8_4f3f_80d7_62a04699cbc5.slice/crio-ce7e56a4e5e7e23e74df41d63074389442fa666be796140e9f761e728dab7a44 WatchSource:0}: Error finding container ce7e56a4e5e7e23e74df41d63074389442fa666be796140e9f761e728dab7a44: Status 404 returned error can't find the container with id ce7e56a4e5e7e23e74df41d63074389442fa666be796140e9f761e728dab7a44
Nov 02 23:57:54.100162 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:54.100122    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:54.386742 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:54.386709    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:54.464060 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:54.464021    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9" event={"ID":"0bf1c293-845f-4179-a74e-8203f5d07479","Type":"ContainerStarted","Data":"d9c52b80ce164d0db4f68710a021c6056ccc7c1c1dca909ba5e437319d3aaca3"}
Nov 02 23:57:54.464284 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:54.464091    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9" event={"ID":"0bf1c293-845f-4179-a74e-8203f5d07479","Type":"ContainerStarted","Data":"8a31b5229b6702faf4b65acd27acd6ad09aa4b3af5055af479f157cf5c22ada3"}
Nov 02 23:57:54.464284 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:54.464130    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9"
Nov 02 23:57:54.466050 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:54.466021    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2" event={"ID":"37c6298e-b0e8-4f3f-80d7-62a04699cbc5","Type":"ContainerStarted","Data":"9a1129f366ab8cbcd05a168945f74feab62cdff928977fdbdf7158b87ea4610a"}
Nov 02 23:57:54.466186 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:54.466063    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2" event={"ID":"37c6298e-b0e8-4f3f-80d7-62a04699cbc5","Type":"ContainerStarted","Data":"ce7e56a4e5e7e23e74df41d63074389442fa666be796140e9f761e728dab7a44"}
Nov 02 23:57:54.466186 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:54.466101    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2"
Nov 02 23:57:54.574821 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:54.574762    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9" podStartSLOduration=1.574748209 podStartE2EDuration="1.574748209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:57:54.573285817 +0000 UTC m=+3914.141640604" watchObservedRunningTime="2026-11-02 23:57:54.574748209 +0000 UTC m=+3914.143102996"
Nov 02 23:57:54.682021 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:54.681873    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2" podStartSLOduration=1.6818532290000001 podStartE2EDuration="1.681853229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:57:54.681151938 +0000 UTC m=+3914.249506725" watchObservedRunningTime="2026-11-02 23:57:54.681853229 +0000 UTC m=+3914.250208018"
Nov 02 23:57:54.978293 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:54.978205    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:56.550169 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:56.550140    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:56.710650 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:56.710618    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:57.511643 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:57.511612    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"]
Nov 02 23:57:57.515702 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:57.515685    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"
Nov 02 23:57:57.569044 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:57.569014    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"]
Nov 02 23:57:57.680765 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:57.680728    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-routing-overlay-volume\") pod \"payload-processing-external-model-696c57bb49-xgkk7\" (UID: \"d9c2fd06-86e4-49d7-9798-7f6face4a9ec\") " pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"
Nov 02 23:57:57.680765 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:57.680768    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9975p\" (UniqueName: \"kubernetes.io/projected/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-kube-api-access-9975p\") pod \"payload-processing-external-model-696c57bb49-xgkk7\" (UID: \"d9c2fd06-86e4-49d7-9798-7f6face4a9ec\") " pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"
Nov 02 23:57:57.681016 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:57.680857    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-plugins-config-volume\") pod \"payload-processing-external-model-696c57bb49-xgkk7\" (UID: \"d9c2fd06-86e4-49d7-9798-7f6face4a9ec\") " pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"
Nov 02 23:57:57.681054 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:57.681035    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-provider-credentials\") pod \"payload-processing-external-model-696c57bb49-xgkk7\" (UID: \"d9c2fd06-86e4-49d7-9798-7f6face4a9ec\") " pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"
Nov 02 23:57:57.782493 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:57.782392    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-plugins-config-volume\") pod \"payload-processing-external-model-696c57bb49-xgkk7\" (UID: \"d9c2fd06-86e4-49d7-9798-7f6face4a9ec\") " pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"
Nov 02 23:57:57.782656 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:57.782548    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-provider-credentials\") pod \"payload-processing-external-model-696c57bb49-xgkk7\" (UID: \"d9c2fd06-86e4-49d7-9798-7f6face4a9ec\") " pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"
Nov 02 23:57:57.782656 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:57.782604    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-routing-overlay-volume\") pod \"payload-processing-external-model-696c57bb49-xgkk7\" (UID: \"d9c2fd06-86e4-49d7-9798-7f6face4a9ec\") " pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"
Nov 02 23:57:57.782656 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:57.782629    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9975p\" (UniqueName: \"kubernetes.io/projected/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-kube-api-access-9975p\") pod \"payload-processing-external-model-696c57bb49-xgkk7\" (UID: \"d9c2fd06-86e4-49d7-9798-7f6face4a9ec\") " pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"
Nov 02 23:57:57.783331 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:57.783295    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-routing-overlay-volume\") pod \"payload-processing-external-model-696c57bb49-xgkk7\" (UID: \"d9c2fd06-86e4-49d7-9798-7f6face4a9ec\") " pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"
Nov 02 23:57:57.783490 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:57.783376    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-plugins-config-volume\") pod \"payload-processing-external-model-696c57bb49-xgkk7\" (UID: \"d9c2fd06-86e4-49d7-9798-7f6face4a9ec\") " pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"
Nov 02 23:57:57.785181 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:57.785160    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-provider-credentials\") pod \"payload-processing-external-model-696c57bb49-xgkk7\" (UID: \"d9c2fd06-86e4-49d7-9798-7f6face4a9ec\") " pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"
Nov 02 23:57:57.836798 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:57.836763    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9975p\" (UniqueName: \"kubernetes.io/projected/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-kube-api-access-9975p\") pod \"payload-processing-external-model-696c57bb49-xgkk7\" (UID: \"d9c2fd06-86e4-49d7-9798-7f6face4a9ec\") " pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"
Nov 02 23:57:58.130054 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:58.129942    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"
Nov 02 23:57:58.498415 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:58.498379    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"]
Nov 02 23:57:58.498687 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:57:58.498666    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c2fd06_86e4_49d7_9798_7f6face4a9ec.slice/crio-e37b6fb172e9bad1b464ef7ac3ffcc2c121e5a857f4f7c3e59acfabab8924697 WatchSource:0}: Error finding container e37b6fb172e9bad1b464ef7ac3ffcc2c121e5a857f4f7c3e59acfabab8924697: Status 404 returned error can't find the container with id e37b6fb172e9bad1b464ef7ac3ffcc2c121e5a857f4f7c3e59acfabab8924697
Nov 02 23:57:58.934338 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:58.934251    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:59.096039 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:59.095996    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:59.501014 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:59.500944    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7" event={"ID":"d9c2fd06-86e4-49d7-9798-7f6face4a9ec","Type":"ContainerStarted","Data":"a5c282d6e5a78a6062721f8c562a1e71cae03b91fb4841aedf4d66d8006997b0"}
Nov 02 23:57:59.501228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:59.501026    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7" event={"ID":"d9c2fd06-86e4-49d7-9798-7f6face4a9ec","Type":"ContainerStarted","Data":"e37b6fb172e9bad1b464ef7ac3ffcc2c121e5a857f4f7c3e59acfabab8924697"}
Nov 02 23:57:59.501228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:59.501142    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"
Nov 02 23:57:59.515857 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:59.515822    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:57:59.611361 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:59.611300    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7" podStartSLOduration=2.611285277 podStartE2EDuration="2.611285277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:57:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:57:59.610095774 +0000 UTC m=+3919.178450562" watchObservedRunningTime="2026-11-02 23:57:59.611285277 +0000 UTC m=+3919.179640063"
Nov 02 23:57:59.688134 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:57:59.688101    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:01.471289 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:01.471251    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:03.917723 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:03.917688    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:04.126923 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:04.126881    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:04.332139 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:04.332105    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:05.406961 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:05.406929    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:05.475088 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:05.475057    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9"
Nov 02 23:58:05.475640 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:05.475622    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2"
Nov 02 23:58:05.571893 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:05.571858    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:08.360305 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.360268    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"]
Nov 02 23:58:08.367228 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.366247    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.413268 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.413237    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh4rh\" (UniqueName: \"kubernetes.io/projected/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-kube-api-access-lh4rh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.413430 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.413287    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.413430 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.413404    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.413500 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.413457    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.413562 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.413546    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.413605 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.413587    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.413637 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.413616    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.413736 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.413717    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.413788 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.413746    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.464491 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.464455    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tbbgz\""
Nov 02 23:58:08.468511 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.468481    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"]
Nov 02 23:58:08.515036 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.514972    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.515202 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.515075    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.515202 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.515123    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.515202 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.515163    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.515202 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.515194    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.515432 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.515224    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.515432 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.515301    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh4rh\" (UniqueName: \"kubernetes.io/projected/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-kube-api-access-lh4rh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.515432 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.515363    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.515596 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.515452    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.515596 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.515508    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.515596 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.515469    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.515818 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.515802    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.516019 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.515969    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.516019 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.516001    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.517804 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.517785    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.517910 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.517893    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.569514 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.569482    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.570135 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.570113    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh4rh\" (UniqueName: \"kubernetes.io/projected/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-kube-api-access-lh4rh\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-6xpg5\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.615277 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.615199    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:08.686192 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.686164    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:08.777155 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.777115    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:08.953403 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:08.953334    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:09.109130 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:58:09.109100    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf40aa5e_4808_47d6_a6c2_3d793e7c0f7c.slice/crio-7549354e367b7a630181c0dd68d933ea9f9a3238b476c11d12b6b83b2e6e9525 WatchSource:0}: Error finding container 7549354e367b7a630181c0dd68d933ea9f9a3238b476c11d12b6b83b2e6e9525: Status 404 returned error can't find the container with id 7549354e367b7a630181c0dd68d933ea9f9a3238b476c11d12b6b83b2e6e9525
Nov 02 23:58:09.109520 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:09.109496    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"]
Nov 02 23:58:09.112008 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:09.111955    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:58:09.112096 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:09.112036    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:58:09.112096 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:09.112064    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:58:09.564992 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:09.564958    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5" event={"ID":"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c","Type":"ContainerStarted","Data":"ab1be8120cb8c17883d692bc2c9f23c9f860b9d6cab443a5d0cae981427995f8"}
Nov 02 23:58:09.565404 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:09.565030    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5" event={"ID":"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c","Type":"ContainerStarted","Data":"7549354e367b7a630181c0dd68d933ea9f9a3238b476c11d12b6b83b2e6e9525"}
Nov 02 23:58:09.681277 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:09.681216    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5" podStartSLOduration=1.6811999100000001 podStartE2EDuration="1.68119991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:58:09.678430976 +0000 UTC m=+3929.246785763" watchObservedRunningTime="2026-11-02 23:58:09.68119991 +0000 UTC m=+3929.249554695"
Nov 02 23:58:09.686852 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:09.686825    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:10.511031 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:10.510964    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7" podUID="d9c2fd06-86e4-49d7-9798-7f6face4a9ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:58:10.688661 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:10.688623    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 02 23:58:10.689101 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:10.688688    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5" podUID="df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 02 23:58:11.686555 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:11.686519    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 02 23:58:11.686728 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:11.686582    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5" podUID="df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 02 23:58:11.751114 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:58:11.751079    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:58:12.687183 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:12.687145    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 02 23:58:12.687361 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:12.687197    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5" podUID="df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 02 23:58:13.348990 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:13.348956    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:13.687089 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:13.686967    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 02 23:58:13.687089 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:13.687046    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5" podUID="df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 02 23:58:14.687096 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:14.687056    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 02 23:58:14.687525 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:14.687110    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5" podUID="df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 02 23:58:15.687439 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:15.687404    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 02 23:58:15.687851 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:15.687459    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5" podUID="df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 02 23:58:16.565450 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:16.565413    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:16.686938 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:16.686893    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 02 23:58:16.687147 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:16.686959    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5" podUID="df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 02 23:58:17.945045 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:17.945007    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:18.523920 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:18.523887    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:18.524370 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:18.524350    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:18.525176 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:18.525159    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 02 23:58:20.509178 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:20.509134    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7" podUID="d9c2fd06-86e4-49d7-9798-7f6face4a9ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:58:22.628079 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:22.628029    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:27.136902 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:27.136855    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:30.508829 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:30.508792    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7" podUID="d9c2fd06-86e4-49d7-9798-7f6face4a9ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:58:31.729153 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:31.729117    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:36.255314 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:36.255281    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:40.508835 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:40.508799    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7" podUID="d9c2fd06-86e4-49d7-9798-7f6face4a9ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:58:40.763699 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:40.763618    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:41.751992 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:58:41.751948    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:58:43.898687 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:43.898640    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:45.442252 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:45.442222    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:45.594118 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:58:45.594087    4294 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb3055757321: secret "agc-cred-eb3055757321" not found
Nov 02 23:58:45.594364 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:58:45.594338    4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-64887bb858-xh446: secret "agc-cred-eb3055757321" not found
Nov 02 23:58:45.594504 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:58:45.594435    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials podName:6ae49deb-156f-4f33-a81d-06bc93b7dcb8 nodeName:}" failed. No retries permitted until 2026-11-02 23:58:46.094411569 +0000 UTC m=+3965.662766350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials") pod "payload-processing-external-model-64887bb858-xh446" (UID: "6ae49deb-156f-4f33-a81d-06bc93b7dcb8") : secret "agc-cred-eb3055757321" not found
Nov 02 23:58:46.099283 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:58:46.099248    4294 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb3055757321: secret "agc-cred-eb3055757321" not found
Nov 02 23:58:46.099283 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:58:46.099285    4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-64887bb858-xh446: secret "agc-cred-eb3055757321" not found
Nov 02 23:58:46.099518 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:58:46.099343    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials podName:6ae49deb-156f-4f33-a81d-06bc93b7dcb8 nodeName:}" failed. No retries permitted until 2026-11-02 23:58:47.099328579 +0000 UTC m=+3966.667683357 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials") pod "payload-processing-external-model-64887bb858-xh446" (UID: "6ae49deb-156f-4f33-a81d-06bc93b7dcb8") : secret "agc-cred-eb3055757321" not found
Nov 02 23:58:47.110182 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:58:47.110152    4294 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb3055757321: secret "agc-cred-eb3055757321" not found
Nov 02 23:58:47.110182 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:58:47.110176    4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-64887bb858-xh446: secret "agc-cred-eb3055757321" not found
Nov 02 23:58:47.110607 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:58:47.110244    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials podName:6ae49deb-156f-4f33-a81d-06bc93b7dcb8 nodeName:}" failed. No retries permitted until 2026-11-02 23:58:49.110231234 +0000 UTC m=+3968.678586000 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials") pod "payload-processing-external-model-64887bb858-xh446" (UID: "6ae49deb-156f-4f33-a81d-06bc93b7dcb8") : secret "agc-cred-eb3055757321" not found
Nov 02 23:58:49.132855 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:58:49.132818    4294 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb3055757321: secret "agc-cred-eb3055757321" not found
Nov 02 23:58:49.132855 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:58:49.132847    4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-64887bb858-xh446: secret "agc-cred-eb3055757321" not found
Nov 02 23:58:49.133342 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:58:49.132901    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials podName:6ae49deb-156f-4f33-a81d-06bc93b7dcb8 nodeName:}" failed. No retries permitted until 2026-11-02 23:58:53.132885408 +0000 UTC m=+3972.701240193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials") pod "payload-processing-external-model-64887bb858-xh446" (UID: "6ae49deb-156f-4f33-a81d-06bc93b7dcb8") : secret "agc-cred-eb3055757321" not found
Nov 02 23:58:49.929547 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:49.929513    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:50.508879 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:50.508839    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7" podUID="d9c2fd06-86e4-49d7-9798-7f6face4a9ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:58:53.179030 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:58:53.178969    4294 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb3055757321: secret "agc-cred-eb3055757321" not found
Nov 02 23:58:53.179030 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:58:53.179023    4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-64887bb858-xh446: secret "agc-cred-eb3055757321" not found
Nov 02 23:58:53.179566 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:58:53.179102    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials podName:6ae49deb-156f-4f33-a81d-06bc93b7dcb8 nodeName:}" failed. No retries permitted until 2026-11-02 23:59:01.179087119 +0000 UTC m=+3980.747441886 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials") pod "payload-processing-external-model-64887bb858-xh446" (UID: "6ae49deb-156f-4f33-a81d-06bc93b7dcb8") : secret "agc-cred-eb3055757321" not found
Nov 02 23:58:54.413609 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:54.413573    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:58.878947 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:58.878886    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:59.040620 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:59.040573    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:58:59.202479 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:58:59.202398    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:00.508617 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:00.508565    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7" podUID="d9c2fd06-86e4-49d7-9798-7f6face4a9ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:59:01.275681 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:59:01.275648    4294 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb3055757321: secret "agc-cred-eb3055757321" not found
Nov 02 23:59:01.275681 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:59:01.275673    4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-64887bb858-xh446: secret "agc-cred-eb3055757321" not found
Nov 02 23:59:01.276026 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:59:01.275748    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials podName:6ae49deb-156f-4f33-a81d-06bc93b7dcb8 nodeName:}" failed. No retries permitted until 2026-11-02 23:59:17.275728973 +0000 UTC m=+3996.844083756 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials") pod "payload-processing-external-model-64887bb858-xh446" (UID: "6ae49deb-156f-4f33-a81d-06bc93b7dcb8") : secret "agc-cred-eb3055757321" not found
Nov 02 23:59:01.542318 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:01.542222    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7" podUID="d9c2fd06-86e4-49d7-9798-7f6face4a9ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:59:03.591179 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:03.591136    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:07.497561 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:07.497527    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"]
Nov 02 23:59:07.498028 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:07.497743    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb" podUID="850d1e3e-f407-4294-9e9a-61471e09b21b" containerName="maas-api" containerID="cri-o://77fdd599357455fe973fddefc55eb3e4fc2d7f11daefed81d0f6ff7380e2ebeb" gracePeriod=30
Nov 02 23:59:07.631217 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:07.631184    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:07.782168 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:07.782136    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"
Nov 02 23:59:07.963103 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:07.963066    4294 generic.go:356] "Generic (PLEG): container finished" podID="850d1e3e-f407-4294-9e9a-61471e09b21b" containerID="77fdd599357455fe973fddefc55eb3e4fc2d7f11daefed81d0f6ff7380e2ebeb" exitCode=0
Nov 02 23:59:07.963287 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:07.963131    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"
Nov 02 23:59:07.963287 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:07.963149    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb" event={"ID":"850d1e3e-f407-4294-9e9a-61471e09b21b","Type":"ContainerDied","Data":"77fdd599357455fe973fddefc55eb3e4fc2d7f11daefed81d0f6ff7380e2ebeb"}
Nov 02 23:59:07.963287 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:07.963235    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb" event={"ID":"850d1e3e-f407-4294-9e9a-61471e09b21b","Type":"ContainerDied","Data":"19cc4bb2c46cf4e1ce02577f1319b746890d2fa459f494174f03146475136880"}
Nov 02 23:59:07.963287 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:07.963257    4294 scope.go:122] "RemoveContainer" containerID="77fdd599357455fe973fddefc55eb3e4fc2d7f11daefed81d0f6ff7380e2ebeb"
Nov 02 23:59:07.966118 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:07.966093    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/850d1e3e-f407-4294-9e9a-61471e09b21b-kube-api-access-96bp7\" (UniqueName: \"kubernetes.io/projected/850d1e3e-f407-4294-9e9a-61471e09b21b-kube-api-access-96bp7\") pod \"850d1e3e-f407-4294-9e9a-61471e09b21b\" (UID: \"850d1e3e-f407-4294-9e9a-61471e09b21b\") "
Nov 02 23:59:07.966266 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:07.966141    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/850d1e3e-f407-4294-9e9a-61471e09b21b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/850d1e3e-f407-4294-9e9a-61471e09b21b-openshift-service-ca\") pod \"850d1e3e-f407-4294-9e9a-61471e09b21b\" (UID: \"850d1e3e-f407-4294-9e9a-61471e09b21b\") "
Nov 02 23:59:07.966266 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:07.966201    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/850d1e3e-f407-4294-9e9a-61471e09b21b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/850d1e3e-f407-4294-9e9a-61471e09b21b-metrics-certs\") pod \"850d1e3e-f407-4294-9e9a-61471e09b21b\" (UID: \"850d1e3e-f407-4294-9e9a-61471e09b21b\") "
Nov 02 23:59:07.966389 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:07.966287    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/850d1e3e-f407-4294-9e9a-61471e09b21b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/850d1e3e-f407-4294-9e9a-61471e09b21b-maas-api-tls\") pod \"850d1e3e-f407-4294-9e9a-61471e09b21b\" (UID: \"850d1e3e-f407-4294-9e9a-61471e09b21b\") "
Nov 02 23:59:07.966592 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:07.966566    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850d1e3e-f407-4294-9e9a-61471e09b21b-openshift-service-ca" pod "850d1e3e-f407-4294-9e9a-61471e09b21b" (UID: "850d1e3e-f407-4294-9e9a-61471e09b21b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:59:07.967038 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:07.967017    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/850d1e3e-f407-4294-9e9a-61471e09b21b-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:59:07.968482 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:07.968455    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850d1e3e-f407-4294-9e9a-61471e09b21b-metrics-certs" pod "850d1e3e-f407-4294-9e9a-61471e09b21b" (UID: "850d1e3e-f407-4294-9e9a-61471e09b21b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:59:07.968659 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:07.968640    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850d1e3e-f407-4294-9e9a-61471e09b21b-kube-api-access-96bp7" pod "850d1e3e-f407-4294-9e9a-61471e09b21b" (UID: "850d1e3e-f407-4294-9e9a-61471e09b21b"). InnerVolumeSpecName "kube-api-access-96bp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:59:07.969806 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:07.969775    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850d1e3e-f407-4294-9e9a-61471e09b21b-maas-api-tls" pod "850d1e3e-f407-4294-9e9a-61471e09b21b" (UID: "850d1e3e-f407-4294-9e9a-61471e09b21b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 23:59:07.977233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:07.977213    4294 scope.go:122] "RemoveContainer" containerID="77fdd599357455fe973fddefc55eb3e4fc2d7f11daefed81d0f6ff7380e2ebeb"
Nov 02 23:59:07.977638 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:59:07.977617    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77fdd599357455fe973fddefc55eb3e4fc2d7f11daefed81d0f6ff7380e2ebeb\": container with ID starting with 77fdd599357455fe973fddefc55eb3e4fc2d7f11daefed81d0f6ff7380e2ebeb not found: ID does not exist" containerID="77fdd599357455fe973fddefc55eb3e4fc2d7f11daefed81d0f6ff7380e2ebeb"
Nov 02 23:59:07.977717 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:07.977649    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77fdd599357455fe973fddefc55eb3e4fc2d7f11daefed81d0f6ff7380e2ebeb"} err="failed to get container status \"77fdd599357455fe973fddefc55eb3e4fc2d7f11daefed81d0f6ff7380e2ebeb\": rpc error: code = NotFound desc = could not find container \"77fdd599357455fe973fddefc55eb3e4fc2d7f11daefed81d0f6ff7380e2ebeb\": container with ID starting with 77fdd599357455fe973fddefc55eb3e4fc2d7f11daefed81d0f6ff7380e2ebeb not found: ID does not exist"
Nov 02 23:59:08.068236 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:08.068148    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96bp7\" (UniqueName: \"kubernetes.io/projected/850d1e3e-f407-4294-9e9a-61471e09b21b-kube-api-access-96bp7\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:59:08.068236 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:08.068177    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/850d1e3e-f407-4294-9e9a-61471e09b21b-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:59:08.068236 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:08.068188    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/850d1e3e-f407-4294-9e9a-61471e09b21b-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:59:08.381114 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:08.381010    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"]
Nov 02 23:59:08.427998 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:08.427958    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a8a1dd-7575f56fc-bvpcb"]
Nov 02 23:59:08.548443 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:08.548408    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="850d1e3e-f407-4294-9e9a-61471e09b21b" path="/var/lib/kubelet/pods/850d1e3e-f407-4294-9e9a-61471e09b21b/volumes"
Nov 02 23:59:09.248106 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:09.248069    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:11.542368 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:11.542327    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7" podUID="d9c2fd06-86e4-49d7-9798-7f6face4a9ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:59:11.754558 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:59:11.754521    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:59:13.698039 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:13.697998    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:13.920422 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:13.920376    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:14.086570 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:14.086535    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:14.768740 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:14.768679    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:14.950185 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:14.950149    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:16.262271 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:16.262230    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:16.748627 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:16.748578    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:17.291584 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:59:17.291544    4294 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb3055757321: secret "agc-cred-eb3055757321" not found
Nov 02 23:59:17.291584 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:59:17.291573    4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-64887bb858-xh446: secret "agc-cred-eb3055757321" not found
Nov 02 23:59:17.292233 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:59:17.291641    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials podName:6ae49deb-156f-4f33-a81d-06bc93b7dcb8 nodeName:}" failed. No retries permitted until 2026-11-02 23:59:49.291625862 +0000 UTC m=+4028.859980632 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials") pod "payload-processing-external-model-64887bb858-xh446" (UID: "6ae49deb-156f-4f33-a81d-06bc93b7dcb8") : secret "agc-cred-eb3055757321" not found
Nov 02 23:59:17.581167 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:17.581071    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:18.168293 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:18.168253    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:18.483830 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:18.483744    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:18.884796 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:18.884755    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:21.542099 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:21.542061    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7" podUID="d9c2fd06-86e4-49d7-9798-7f6face4a9ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:59:22.377844 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:22.377806    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:22.585001 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:22.582569    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:23.855190 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:23.855104    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:25.525842 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:25.525801    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:25.684045 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:25.684008    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:25.912126 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:25.912038    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2"]
Nov 02 23:59:25.912718 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:25.912704    4294 cpu_manager.go:414] "containerMap: removing container" podUID="850d1e3e-f407-4294-9e9a-61471e09b21b" containerName="maas-api"
Nov 02 23:59:25.912765 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:25.912719    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="850d1e3e-f407-4294-9e9a-61471e09b21b" containerName="maas-api"
Nov 02 23:59:25.912850 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:25.912842    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="850d1e3e-f407-4294-9e9a-61471e09b21b" containerName="maas-api"
Nov 02 23:59:25.916181 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:25.916164    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2"
Nov 02 23:59:25.968417 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:25.968387    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2"]
Nov 02 23:59:26.012337 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:26.012302    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-68d89f\""
Nov 02 23:59:26.012566 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:26.012365    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-68d89f-dockercfg-ngvj5\""
Nov 02 23:59:26.047049 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:26.047012    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75f869b3-35bd-4168-bdd5-8a6334bc8e95-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2\" (UID: \"75f869b3-35bd-4168-bdd5-8a6334bc8e95\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2"
Nov 02 23:59:26.047253 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:26.047225    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xx7p\" (UniqueName: \"kubernetes.io/projected/75f869b3-35bd-4168-bdd5-8a6334bc8e95-kube-api-access-5xx7p\") pod \"payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2\" (UID: \"75f869b3-35bd-4168-bdd5-8a6334bc8e95\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2"
Nov 02 23:59:26.148325 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:26.148288    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75f869b3-35bd-4168-bdd5-8a6334bc8e95-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2\" (UID: \"75f869b3-35bd-4168-bdd5-8a6334bc8e95\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2"
Nov 02 23:59:26.148514 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:26.148436    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xx7p\" (UniqueName: \"kubernetes.io/projected/75f869b3-35bd-4168-bdd5-8a6334bc8e95-kube-api-access-5xx7p\") pod \"payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2\" (UID: \"75f869b3-35bd-4168-bdd5-8a6334bc8e95\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2"
Nov 02 23:59:26.149053 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:26.149034    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75f869b3-35bd-4168-bdd5-8a6334bc8e95-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2\" (UID: \"75f869b3-35bd-4168-bdd5-8a6334bc8e95\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2"
Nov 02 23:59:26.203085 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:26.203010    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xx7p\" (UniqueName: \"kubernetes.io/projected/75f869b3-35bd-4168-bdd5-8a6334bc8e95-kube-api-access-5xx7p\") pod \"payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2\" (UID: \"75f869b3-35bd-4168-bdd5-8a6334bc8e95\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2"
Nov 02 23:59:26.230842 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:26.230811    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2"
Nov 02 23:59:26.597632 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:26.597604    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2"]
Nov 02 23:59:26.598659 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:59:26.598629    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f869b3_35bd_4168_bdd5_8a6334bc8e95.slice/crio-ca6b58aa5a9b06399420f9e7415503938376d8955c6324c480e644ae7010fd08 WatchSource:0}: Error finding container ca6b58aa5a9b06399420f9e7415503938376d8955c6324c480e644ae7010fd08: Status 404 returned error can't find the container with id ca6b58aa5a9b06399420f9e7415503938376d8955c6324c480e644ae7010fd08
Nov 02 23:59:27.129414 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:27.129369    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2" event={"ID":"75f869b3-35bd-4168-bdd5-8a6334bc8e95","Type":"ContainerStarted","Data":"4165c188da3d3d0c72c1f2bd68399b92acc653a952b679fa612593db5b3f8e47"}
Nov 02 23:59:27.129614 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:27.129444    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2"
Nov 02 23:59:27.129614 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:27.129468    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2" event={"ID":"75f869b3-35bd-4168-bdd5-8a6334bc8e95","Type":"ContainerStarted","Data":"ca6b58aa5a9b06399420f9e7415503938376d8955c6324c480e644ae7010fd08"}
Nov 02 23:59:27.239193 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:27.239134    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2" podStartSLOduration=2.23911862 podStartE2EDuration="2.23911862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:59:27.237947286 +0000 UTC m=+4006.806302074" watchObservedRunningTime="2026-11-02 23:59:27.23911862 +0000 UTC m=+4006.807473468"
Nov 02 23:59:28.626523 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:28.626484    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:31.542064 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:31.542029    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7" podUID="d9c2fd06-86e4-49d7-9798-7f6face4a9ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:59:33.257657 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:33.257621    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:37.724914 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:37.724875    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:38.140221 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:38.140192    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2"
Nov 02 23:59:39.067329 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.067286    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"]
Nov 02 23:59:39.073041 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.073014    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.218549 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.218502    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f60ec698-0068-4422-aff1-c8ecbe35f709-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.218741 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.218615    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60ec698-0068-4422-aff1-c8ecbe35f709-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.218741 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.218652    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f60ec698-0068-4422-aff1-c8ecbe35f709-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.218741 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.218689    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f60ec698-0068-4422-aff1-c8ecbe35f709-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.218856 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.218771    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7d5g\" (UniqueName: \"kubernetes.io/projected/f60ec698-0068-4422-aff1-c8ecbe35f709-kube-api-access-m7d5g\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.218856 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.218790    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f60ec698-0068-4422-aff1-c8ecbe35f709-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.218953 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.218932    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f60ec698-0068-4422-aff1-c8ecbe35f709-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.219078 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.219054    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60ec698-0068-4422-aff1-c8ecbe35f709-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.219191 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.219174    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f60ec698-0068-4422-aff1-c8ecbe35f709-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.267283 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.267247    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"]
Nov 02 23:59:39.320531 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.320437    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60ec698-0068-4422-aff1-c8ecbe35f709-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.320696 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.320538    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f60ec698-0068-4422-aff1-c8ecbe35f709-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.320696 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.320627    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f60ec698-0068-4422-aff1-c8ecbe35f709-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.320696 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.320673    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60ec698-0068-4422-aff1-c8ecbe35f709-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.320943 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.320712    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f60ec698-0068-4422-aff1-c8ecbe35f709-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.320943 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.320739    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f60ec698-0068-4422-aff1-c8ecbe35f709-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.320943 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.320812    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7d5g\" (UniqueName: \"kubernetes.io/projected/f60ec698-0068-4422-aff1-c8ecbe35f709-kube-api-access-m7d5g\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.320943 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.320837    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f60ec698-0068-4422-aff1-c8ecbe35f709-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.320943 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.320923    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60ec698-0068-4422-aff1-c8ecbe35f709-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.321256 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.320950    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f60ec698-0068-4422-aff1-c8ecbe35f709-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.321256 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.321082    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f60ec698-0068-4422-aff1-c8ecbe35f709-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.321256 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.321163    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60ec698-0068-4422-aff1-c8ecbe35f709-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.321680 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.321654    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f60ec698-0068-4422-aff1-c8ecbe35f709-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.321932 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.321904    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f60ec698-0068-4422-aff1-c8ecbe35f709-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.323050 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.323029    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f60ec698-0068-4422-aff1-c8ecbe35f709-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.323375 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.323352    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f60ec698-0068-4422-aff1-c8ecbe35f709-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.375369 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.375337    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f60ec698-0068-4422-aff1-c8ecbe35f709-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.375589 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.375564    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7d5g\" (UniqueName: \"kubernetes.io/projected/f60ec698-0068-4422-aff1-c8ecbe35f709-kube-api-access-m7d5g\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-lgl4w\" (UID: \"f60ec698-0068-4422-aff1-c8ecbe35f709\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.388007 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.387966    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:39.808901 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.808862    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"]
Nov 02 23:59:39.844459 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:59:39.844428    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60ec698_0068_4422_aff1_c8ecbe35f709.slice/crio-791596393e8f4889ca62ccb992bae422cf7c46b7c08053c33118eeb3c9f69e20 WatchSource:0}: Error finding container 791596393e8f4889ca62ccb992bae422cf7c46b7c08053c33118eeb3c9f69e20: Status 404 returned error can't find the container with id 791596393e8f4889ca62ccb992bae422cf7c46b7c08053c33118eeb3c9f69e20
Nov 02 23:59:39.847145 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.847109    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:59:39.847257 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.847184    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:59:39.847257 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:39.847213    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 23:59:40.215116 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:40.215026    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w" event={"ID":"f60ec698-0068-4422-aff1-c8ecbe35f709","Type":"ContainerStarted","Data":"7c1062ab8ce634d292429d9420f6f3aded790f8cbe4a5922e22d7597aff2a191"}
Nov 02 23:59:40.215116 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:40.215087    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w" event={"ID":"f60ec698-0068-4422-aff1-c8ecbe35f709","Type":"ContainerStarted","Data":"791596393e8f4889ca62ccb992bae422cf7c46b7c08053c33118eeb3c9f69e20"}
Nov 02 23:59:40.266036 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:40.266003    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:40.330172 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:40.330104    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w" podStartSLOduration=1.330082697 podStartE2EDuration="1.330082697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:59:40.32873483 +0000 UTC m=+4019.897089617" watchObservedRunningTime="2026-11-02 23:59:40.330082697 +0000 UTC m=+4019.898437484"
Nov 02 23:59:40.388358 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:40.388322    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:40.390046 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:40.390018    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 02 23:59:40.390201 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:40.390069    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w" podUID="f60ec698-0068-4422-aff1-c8ecbe35f709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 02 23:59:40.443604 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:40.443567    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:41.224524 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.224491    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"]
Nov 02 23:59:41.225854 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.225820    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7" podUID="d9c2fd06-86e4-49d7-9798-7f6face4a9ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:59:41.279736 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.279699    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-759489599d-ft2jp"]
Nov 02 23:59:41.284531 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.284497    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp"
Nov 02 23:59:41.311909 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.311878    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-759489599d-ft2jp"]
Nov 02 23:59:41.349895 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.349859    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-routing-overlay-volume\") pod \"payload-processing-external-model-759489599d-ft2jp\" (UID: \"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593\") " pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp"
Nov 02 23:59:41.350101 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.350080    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r79jc\" (UniqueName: \"kubernetes.io/projected/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-kube-api-access-r79jc\") pod \"payload-processing-external-model-759489599d-ft2jp\" (UID: \"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593\") " pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp"
Nov 02 23:59:41.350243 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.350221    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-provider-credentials\") pod \"payload-processing-external-model-759489599d-ft2jp\" (UID: \"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593\") " pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp"
Nov 02 23:59:41.350304 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.350273    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-plugins-config-volume\") pod \"payload-processing-external-model-759489599d-ft2jp\" (UID: \"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593\") " pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp"
Nov 02 23:59:41.388405 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.388371    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 02 23:59:41.388598 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.388423    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w" podUID="f60ec698-0068-4422-aff1-c8ecbe35f709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 02 23:59:41.451877 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.451843    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-plugins-config-volume\") pod \"payload-processing-external-model-759489599d-ft2jp\" (UID: \"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593\") " pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp"
Nov 02 23:59:41.452084 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.451960    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-routing-overlay-volume\") pod \"payload-processing-external-model-759489599d-ft2jp\" (UID: \"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593\") " pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp"
Nov 02 23:59:41.452136 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.452110    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r79jc\" (UniqueName: \"kubernetes.io/projected/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-kube-api-access-r79jc\") pod \"payload-processing-external-model-759489599d-ft2jp\" (UID: \"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593\") " pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp"
Nov 02 23:59:41.452233 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.452215    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-provider-credentials\") pod \"payload-processing-external-model-759489599d-ft2jp\" (UID: \"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593\") " pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp"
Nov 02 23:59:41.452630 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.452605    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-routing-overlay-volume\") pod \"payload-processing-external-model-759489599d-ft2jp\" (UID: \"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593\") " pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp"
Nov 02 23:59:41.452630 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.452616    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-plugins-config-volume\") pod \"payload-processing-external-model-759489599d-ft2jp\" (UID: \"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593\") " pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp"
Nov 02 23:59:41.478845 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.478757    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-provider-credentials\") pod \"payload-processing-external-model-759489599d-ft2jp\" (UID: \"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593\") " pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp"
Nov 02 23:59:41.506692 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.506654    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r79jc\" (UniqueName: \"kubernetes.io/projected/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-kube-api-access-r79jc\") pod \"payload-processing-external-model-759489599d-ft2jp\" (UID: \"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593\") " pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp"
Nov 02 23:59:41.599518 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.599483    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp"
Nov 02 23:59:41.785731 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:59:41.785687    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 02 23:59:41.990364 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:59:41.990298    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc5dfb6_80a9_4f88_9ca0_e0f450e3a593.slice/crio-04fb8408f602d738081cdb3e270a14df4e5ed626e27ba6cf8433f48367936f89 WatchSource:0}: Error finding container 04fb8408f602d738081cdb3e270a14df4e5ed626e27ba6cf8433f48367936f89: Status 404 returned error can't find the container with id 04fb8408f602d738081cdb3e270a14df4e5ed626e27ba6cf8433f48367936f89
Nov 02 23:59:41.990914 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:41.990865    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-759489599d-ft2jp"]
Nov 02 23:59:42.230520 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:42.230478    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" event={"ID":"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593","Type":"ContainerStarted","Data":"1b49f11572f20432a37be02bbea357167c7e53eee93fc85e17ea5f8576287c2d"}
Nov 02 23:59:42.231056 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:42.230548    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" event={"ID":"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593","Type":"ContainerStarted","Data":"04fb8408f602d738081cdb3e270a14df4e5ed626e27ba6cf8433f48367936f89"}
Nov 02 23:59:42.231056 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:42.230586    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp"
Nov 02 23:59:42.285093 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:42.285059    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:42.340351 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:42.340294    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podStartSLOduration=1.340279766 podStartE2EDuration="1.340279766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:59:42.339678675 +0000 UTC m=+4021.908033464" watchObservedRunningTime="2026-11-02 23:59:42.340279766 +0000 UTC m=+4021.908634592"
Nov 02 23:59:42.389153 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:42.389122    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 02 23:59:42.389333 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:42.389179    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w" podUID="f60ec698-0068-4422-aff1-c8ecbe35f709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 02 23:59:42.479584 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:42.479546    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:42.878760 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:42.878606    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:43.307962 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:43.307919    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:43.388494 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:43.388448    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 02 23:59:43.388683 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:43.388523    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w" podUID="f60ec698-0068-4422-aff1-c8ecbe35f709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 02 23:59:44.285477 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:44.285423    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:44.388725 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:44.388686    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 02 23:59:44.389202 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:44.388745    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w" podUID="f60ec698-0068-4422-aff1-c8ecbe35f709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 02 23:59:45.389361 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:45.389319    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 02 23:59:45.389783 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:45.389389    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w" podUID="f60ec698-0068-4422-aff1-c8ecbe35f709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 02 23:59:46.225584 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:46.225534    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7" podUID="d9c2fd06-86e4-49d7-9798-7f6face4a9ec" containerName="payload-processing" containerID="cri-o://a5c282d6e5a78a6062721f8c562a1e71cae03b91fb4841aedf4d66d8006997b0" gracePeriod=25
Nov 02 23:59:46.390023 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:46.389022    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 02 23:59:46.390023 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:46.389102    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w" podUID="f60ec698-0068-4422-aff1-c8ecbe35f709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 02 23:59:46.429170 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:46.429100    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:46.499098 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:46.499053    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"
Nov 02 23:59:46.543119 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:46.543081    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-provider-credentials\") pod \"d9c2fd06-86e4-49d7-9798-7f6face4a9ec\" (UID: \"d9c2fd06-86e4-49d7-9798-7f6face4a9ec\") "
Nov 02 23:59:46.543365 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:46.543258    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-routing-overlay-volume\") pod \"d9c2fd06-86e4-49d7-9798-7f6face4a9ec\" (UID: \"d9c2fd06-86e4-49d7-9798-7f6face4a9ec\") "
Nov 02 23:59:46.543452 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:46.543365    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-kube-api-access-9975p\" (UniqueName: \"kubernetes.io/projected/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-kube-api-access-9975p\") pod \"d9c2fd06-86e4-49d7-9798-7f6face4a9ec\" (UID: \"d9c2fd06-86e4-49d7-9798-7f6face4a9ec\") "
Nov 02 23:59:46.543452 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:46.543398    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-plugins-config-volume\") pod \"d9c2fd06-86e4-49d7-9798-7f6face4a9ec\" (UID: \"d9c2fd06-86e4-49d7-9798-7f6face4a9ec\") "
Nov 02 23:59:46.544973 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:46.543722    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-routing-overlay-volume" pod "d9c2fd06-86e4-49d7-9798-7f6face4a9ec" (UID: "d9c2fd06-86e4-49d7-9798-7f6face4a9ec"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:59:46.544973 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:46.544047    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-plugins-config-volume" pod "d9c2fd06-86e4-49d7-9798-7f6face4a9ec" (UID: "d9c2fd06-86e4-49d7-9798-7f6face4a9ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 23:59:46.544973 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:46.544459    4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-routing-overlay-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:59:46.544973 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:46.544494    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:59:46.545777 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:46.545747    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-provider-credentials" pod "d9c2fd06-86e4-49d7-9798-7f6face4a9ec" (UID: "d9c2fd06-86e4-49d7-9798-7f6face4a9ec"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:59:46.545907 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:46.545884    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-kube-api-access-9975p" pod "d9c2fd06-86e4-49d7-9798-7f6face4a9ec" (UID: "d9c2fd06-86e4-49d7-9798-7f6face4a9ec"). InnerVolumeSpecName "kube-api-access-9975p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 23:59:46.645746 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:46.645470    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9975p\" (UniqueName: \"kubernetes.io/projected/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-kube-api-access-9975p\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:59:46.645746 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:46.645503    4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d9c2fd06-86e4-49d7-9798-7f6face4a9ec-provider-credentials\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 02 23:59:46.732310 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:46.732211    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:46.899861 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:46.899810    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:47.202194 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:47.202151    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:47.267852 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:47.267819    4294 generic.go:356] "Generic (PLEG): container finished" podID="d9c2fd06-86e4-49d7-9798-7f6face4a9ec" containerID="a5c282d6e5a78a6062721f8c562a1e71cae03b91fb4841aedf4d66d8006997b0" exitCode=0
Nov 02 23:59:47.268075 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:47.267886    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"
Nov 02 23:59:47.268075 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:47.267910    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7" event={"ID":"d9c2fd06-86e4-49d7-9798-7f6face4a9ec","Type":"ContainerDied","Data":"a5c282d6e5a78a6062721f8c562a1e71cae03b91fb4841aedf4d66d8006997b0"}
Nov 02 23:59:47.268075 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:47.267964    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7" event={"ID":"d9c2fd06-86e4-49d7-9798-7f6face4a9ec","Type":"ContainerDied","Data":"e37b6fb172e9bad1b464ef7ac3ffcc2c121e5a857f4f7c3e59acfabab8924697"}
Nov 02 23:59:47.268075 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:47.267997    4294 scope.go:122] "RemoveContainer" containerID="a5c282d6e5a78a6062721f8c562a1e71cae03b91fb4841aedf4d66d8006997b0"
Nov 02 23:59:47.277939 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:47.277913    4294 scope.go:122] "RemoveContainer" containerID="a5c282d6e5a78a6062721f8c562a1e71cae03b91fb4841aedf4d66d8006997b0"
Nov 02 23:59:47.278357 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:59:47.278328    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c282d6e5a78a6062721f8c562a1e71cae03b91fb4841aedf4d66d8006997b0\": container with ID starting with a5c282d6e5a78a6062721f8c562a1e71cae03b91fb4841aedf4d66d8006997b0 not found: ID does not exist" containerID="a5c282d6e5a78a6062721f8c562a1e71cae03b91fb4841aedf4d66d8006997b0"
Nov 02 23:59:47.278446 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:47.278364    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c282d6e5a78a6062721f8c562a1e71cae03b91fb4841aedf4d66d8006997b0"} err="failed to get container status \"a5c282d6e5a78a6062721f8c562a1e71cae03b91fb4841aedf4d66d8006997b0\": rpc error: code = NotFound desc = could not find container \"a5c282d6e5a78a6062721f8c562a1e71cae03b91fb4841aedf4d66d8006997b0\": container with ID starting with a5c282d6e5a78a6062721f8c562a1e71cae03b91fb4841aedf4d66d8006997b0 not found: ID does not exist"
Nov 02 23:59:47.389223 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:47.389177    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 02 23:59:47.389399 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:47.389255    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w" podUID="f60ec698-0068-4422-aff1-c8ecbe35f709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 02 23:59:47.427047 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:47.427004    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"]
Nov 02 23:59:47.474479 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:47.474397    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-696c57bb49-xgkk7"]
Nov 02 23:59:48.549465 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:48.549426    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9c2fd06-86e4-49d7-9798-7f6face4a9ec" path="/var/lib/kubelet/pods/d9c2fd06-86e4-49d7-9798-7f6face4a9ec/volumes"
Nov 02 23:59:48.715138 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:48.715101    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:48.715624 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:48.715601    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:48.716605 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:48.716585    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w"
Nov 02 23:59:49.387380 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:59:49.387337    4294 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a610cb5fdd81: secret "agc-cred-a610cb5fdd81" not found
Nov 02 23:59:49.387380 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:59:49.387373    4294 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb3055757321: secret "agc-cred-eb3055757321" not found
Nov 02 23:59:49.387380 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:59:49.387387    4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-64887bb858-xh446: [secret "agc-cred-a610cb5fdd81" not found, secret "agc-cred-eb3055757321" not found]
Nov 02 23:59:49.387715 ip-10-0-139-234 kubenswrapper[4294]: E1102 23:59:49.387448    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials podName:6ae49deb-156f-4f33-a81d-06bc93b7dcb8 nodeName:}" failed. No retries permitted until 2026-11-03 00:00:53.387430009 +0000 UTC m=+4092.955784788 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials") pod "payload-processing-external-model-64887bb858-xh446" (UID: "6ae49deb-156f-4f33-a81d-06bc93b7dcb8") : [secret "agc-cred-a610cb5fdd81" not found, secret "agc-cred-eb3055757321" not found]
Nov 02 23:59:49.543012 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:49.542968    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:49.711963 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:49.711852    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:49.923689 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:49.923652    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6"]
Nov 02 23:59:49.924303 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:49.924288    4294 cpu_manager.go:414] "containerMap: removing container" podUID="d9c2fd06-86e4-49d7-9798-7f6face4a9ec" containerName="payload-processing"
Nov 02 23:59:49.924357 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:49.924303    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c2fd06-86e4-49d7-9798-7f6face4a9ec" containerName="payload-processing"
Nov 02 23:59:49.924444 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:49.924434    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c2fd06-86e4-49d7-9798-7f6face4a9ec" containerName="payload-processing"
Nov 02 23:59:49.927870 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:49.927853    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6"
Nov 02 23:59:49.980524 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:49.980450    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6"]
Nov 02 23:59:49.994957 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:49.994909    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb92a5d9-abf6-4d20-9919-5edce78d957f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6\" (UID: \"bb92a5d9-abf6-4d20-9919-5edce78d957f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6"
Nov 02 23:59:49.995145 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:49.995112    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qqnn\" (UniqueName: \"kubernetes.io/projected/bb92a5d9-abf6-4d20-9919-5edce78d957f-kube-api-access-2qqnn\") pod \"payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6\" (UID: \"bb92a5d9-abf6-4d20-9919-5edce78d957f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6"
Nov 02 23:59:50.023780 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:50.023741    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1f29e6\""
Nov 02 23:59:50.024052 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:50.024016    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1f29e6-dockercfg-lnsf5\""
Nov 02 23:59:50.099091 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:50.099041    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj"]
Nov 02 23:59:50.099574 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:50.099493    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb92a5d9-abf6-4d20-9919-5edce78d957f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6\" (UID: \"bb92a5d9-abf6-4d20-9919-5edce78d957f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6"
Nov 02 23:59:50.099675 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:50.099629    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qqnn\" (UniqueName: \"kubernetes.io/projected/bb92a5d9-abf6-4d20-9919-5edce78d957f-kube-api-access-2qqnn\") pod \"payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6\" (UID: \"bb92a5d9-abf6-4d20-9919-5edce78d957f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6"
Nov 02 23:59:50.100888 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:50.100855    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb92a5d9-abf6-4d20-9919-5edce78d957f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6\" (UID: \"bb92a5d9-abf6-4d20-9919-5edce78d957f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6"
Nov 02 23:59:50.107083 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:50.107055    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj"
Nov 02 23:59:50.150602 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:50.150569    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj"]
Nov 02 23:59:50.154914 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:50.154882    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qqnn\" (UniqueName: \"kubernetes.io/projected/bb92a5d9-abf6-4d20-9919-5edce78d957f-kube-api-access-2qqnn\") pod \"payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6\" (UID: \"bb92a5d9-abf6-4d20-9919-5edce78d957f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6"
Nov 02 23:59:50.200860 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:50.200816    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d36c485f-2e64-48e6-ae55-df84f85dd47b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj\" (UID: \"d36c485f-2e64-48e6-ae55-df84f85dd47b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj"
Nov 02 23:59:50.201051 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:50.200888    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndjvc\" (UniqueName: \"kubernetes.io/projected/d36c485f-2e64-48e6-ae55-df84f85dd47b-kube-api-access-ndjvc\") pod \"payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj\" (UID: \"d36c485f-2e64-48e6-ae55-df84f85dd47b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj"
Nov 02 23:59:50.241481 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:50.241453    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6"
Nov 02 23:59:50.303053 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:50.303026    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d36c485f-2e64-48e6-ae55-df84f85dd47b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj\" (UID: \"d36c485f-2e64-48e6-ae55-df84f85dd47b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj"
Nov 02 23:59:50.303202 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:50.303096    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndjvc\" (UniqueName: \"kubernetes.io/projected/d36c485f-2e64-48e6-ae55-df84f85dd47b-kube-api-access-ndjvc\") pod \"payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj\" (UID: \"d36c485f-2e64-48e6-ae55-df84f85dd47b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj"
Nov 02 23:59:50.303950 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:50.303923    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d36c485f-2e64-48e6-ae55-df84f85dd47b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj\" (UID: \"d36c485f-2e64-48e6-ae55-df84f85dd47b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj"
Nov 02 23:59:50.358021 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:50.357952    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndjvc\" (UniqueName: \"kubernetes.io/projected/d36c485f-2e64-48e6-ae55-df84f85dd47b-kube-api-access-ndjvc\") pod \"payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj\" (UID: \"d36c485f-2e64-48e6-ae55-df84f85dd47b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj"
Nov 02 23:59:50.421538 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:50.421503    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj"
Nov 02 23:59:50.611728 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:59:50.611697    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb92a5d9_abf6_4d20_9919_5edce78d957f.slice/crio-0f35a027358f568c94787225c77475da971000d6599fc65fef245e7aea8c75df WatchSource:0}: Error finding container 0f35a027358f568c94787225c77475da971000d6599fc65fef245e7aea8c75df: Status 404 returned error can't find the container with id 0f35a027358f568c94787225c77475da971000d6599fc65fef245e7aea8c75df
Nov 02 23:59:50.613179 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:50.613151    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6"]
Nov 02 23:59:50.791364 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:50.791322    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj"]
Nov 02 23:59:50.791812 ip-10-0-139-234 kubenswrapper[4294]: W1102 23:59:50.791509    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36c485f_2e64_48e6_ae55_df84f85dd47b.slice/crio-26c86a6c72b7a33523fb5168dfb1976bc11ba2e3cd57e6737a9e687b88a57ae3 WatchSource:0}: Error finding container 26c86a6c72b7a33523fb5168dfb1976bc11ba2e3cd57e6737a9e687b88a57ae3: Status 404 returned error can't find the container with id 26c86a6c72b7a33523fb5168dfb1976bc11ba2e3cd57e6737a9e687b88a57ae3
Nov 02 23:59:51.299522 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:51.299486    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj" event={"ID":"d36c485f-2e64-48e6-ae55-df84f85dd47b","Type":"ContainerStarted","Data":"83ff5d2d98bdddd2b77e21c42c75b6861ad91e3c270563ace06302b936b27593"}
Nov 02 23:59:51.299744 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:51.299542    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj" event={"ID":"d36c485f-2e64-48e6-ae55-df84f85dd47b","Type":"ContainerStarted","Data":"26c86a6c72b7a33523fb5168dfb1976bc11ba2e3cd57e6737a9e687b88a57ae3"}
Nov 02 23:59:51.299744 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:51.299572    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj"
Nov 02 23:59:51.301334 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:51.301308    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6" event={"ID":"bb92a5d9-abf6-4d20-9919-5edce78d957f","Type":"ContainerStarted","Data":"05c49aa21512b56e110ff6513234ebfe9026482b3d3c875d5cde1cf9edfda65b"}
Nov 02 23:59:51.301471 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:51.301358    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6" event={"ID":"bb92a5d9-abf6-4d20-9919-5edce78d957f","Type":"ContainerStarted","Data":"0f35a027358f568c94787225c77475da971000d6599fc65fef245e7aea8c75df"}
Nov 02 23:59:51.301471 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:51.301388    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6"
Nov 02 23:59:51.408822 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:51.408747    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj" podStartSLOduration=1.408729775 podStartE2EDuration="1.408729775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:59:51.407934085 +0000 UTC m=+4030.976288873" watchObservedRunningTime="2026-11-02 23:59:51.408729775 +0000 UTC m=+4030.977084559"
Nov 02 23:59:51.516894 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:51.516840    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6" podStartSLOduration=2.516822732 podStartE2EDuration="2.516822732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 23:59:51.515515306 +0000 UTC m=+4031.083870082" watchObservedRunningTime="2026-11-02 23:59:51.516822732 +0000 UTC m=+4031.085177561"
Nov 02 23:59:51.947044 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:51.947009    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:53.243886 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:53.243846    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 23:59:54.136622 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:54.136582    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:54.301246 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:54.301204    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 02 23:59:56.444642 ip-10-0-139-234 kubenswrapper[4294]: I1102 23:59:56.444602    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:00.170203 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.170168    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29894400-vhdh5"]
Nov 03 00:00:00.175664 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.175636    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29894400-vhdh5"
Nov 03 00:00:00.196964 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.196924    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8"]
Nov 03 00:00:00.205432 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.205397    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8"
Nov 03 00:00:00.210569 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.210537    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft"]
Nov 03 00:00:00.217400 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.217378    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft"
Nov 03 00:00:00.234376 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.234348    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\""
Nov 03 00:00:00.241774 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.239673    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-mrhsc\""
Nov 03 00:00:00.245794 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.245772    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx"]
Nov 03 00:00:00.251272 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.251251    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht"]
Nov 03 00:00:00.251533 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.251515    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx"
Nov 03 00:00:00.255585 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.255559    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht"
Nov 03 00:00:00.283814 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.283787    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29894400-vhdh5"]
Nov 03 00:00:00.291263 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.291235    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv"]
Nov 03 00:00:00.300245 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.298011    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv"
Nov 03 00:00:00.332561 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.332459    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft"]
Nov 03 00:00:00.333187 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.333156    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dtgr\" (UniqueName: \"kubernetes.io/projected/8b200fdb-64bb-40f0-9663-1d7f1e7e3723-kube-api-access-5dtgr\") pod \"maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx\" (UID: \"8b200fdb-64bb-40f0-9663-1d7f1e7e3723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx"
Nov 03 00:00:00.333451 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.333423    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c95f99-51cf-4420-bb11-d6022fd906f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht\" (UID: \"d9c95f99-51cf-4420-bb11-d6022fd906f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht"
Nov 03 00:00:00.333584 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.333468    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzrlw\" (UniqueName: \"kubernetes.io/projected/d9c95f99-51cf-4420-bb11-d6022fd906f4-kube-api-access-wzrlw\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht\" (UID: \"d9c95f99-51cf-4420-bb11-d6022fd906f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht"
Nov 03 00:00:00.333584 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.333502    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wxpb\" (UniqueName: \"kubernetes.io/projected/a8c748df-cfe2-4382-b65c-dbf9937c03d9-kube-api-access-6wxpb\") pod \"image-pruner-29894400-vhdh5\" (UID: \"a8c748df-cfe2-4382-b65c-dbf9937c03d9\") " pod="openshift-image-registry/image-pruner-29894400-vhdh5"
Nov 03 00:00:00.333584 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.333560    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfxwl\" (UniqueName: \"kubernetes.io/projected/4ecf1fa4-ede5-420c-ace8-ef7bfb01423b-kube-api-access-zfxwl\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8\" (UID: \"4ecf1fa4-ede5-420c-ace8-ef7bfb01423b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8"
Nov 03 00:00:00.333782 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.333612    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e595d849-e5c1-49db-90f3-dc35a24158bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft\" (UID: \"e595d849-e5c1-49db-90f3-dc35a24158bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft"
Nov 03 00:00:00.333782 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.333717    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6xbb\" (UniqueName: \"kubernetes.io/projected/e595d849-e5c1-49db-90f3-dc35a24158bd-kube-api-access-s6xbb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft\" (UID: \"e595d849-e5c1-49db-90f3-dc35a24158bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft"
Nov 03 00:00:00.333902 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.333781    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ecf1fa4-ede5-420c-ace8-ef7bfb01423b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8\" (UID: \"4ecf1fa4-ede5-420c-ace8-ef7bfb01423b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8"
Nov 03 00:00:00.333902 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.333832    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b200fdb-64bb-40f0-9663-1d7f1e7e3723-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx\" (UID: \"8b200fdb-64bb-40f0-9663-1d7f1e7e3723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx"
Nov 03 00:00:00.333902 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.333864    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8c748df-cfe2-4382-b65c-dbf9937c03d9-serviceca\") pod \"image-pruner-29894400-vhdh5\" (UID: \"a8c748df-cfe2-4382-b65c-dbf9937c03d9\") " pod="openshift-image-registry/image-pruner-29894400-vhdh5"
Nov 03 00:00:00.337692 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.337672    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\""
Nov 03 00:00:00.339947 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.339918    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8"]
Nov 03 00:00:00.378088 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.378048    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht"]
Nov 03 00:00:00.381594 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.381512    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv"]
Nov 03 00:00:00.389600 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.389556    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx"]
Nov 03 00:00:00.435060 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.434934    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dtgr\" (UniqueName: \"kubernetes.io/projected/8b200fdb-64bb-40f0-9663-1d7f1e7e3723-kube-api-access-5dtgr\") pod \"maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx\" (UID: \"8b200fdb-64bb-40f0-9663-1d7f1e7e3723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx"
Nov 03 00:00:00.435060 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.435023    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c95f99-51cf-4420-bb11-d6022fd906f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht\" (UID: \"d9c95f99-51cf-4420-bb11-d6022fd906f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht"
Nov 03 00:00:00.435060 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.435050    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzrlw\" (UniqueName: \"kubernetes.io/projected/d9c95f99-51cf-4420-bb11-d6022fd906f4-kube-api-access-wzrlw\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht\" (UID: \"d9c95f99-51cf-4420-bb11-d6022fd906f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht"
Nov 03 00:00:00.435373 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.435069    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wxpb\" (UniqueName: \"kubernetes.io/projected/a8c748df-cfe2-4382-b65c-dbf9937c03d9-kube-api-access-6wxpb\") pod \"image-pruner-29894400-vhdh5\" (UID: \"a8c748df-cfe2-4382-b65c-dbf9937c03d9\") " pod="openshift-image-registry/image-pruner-29894400-vhdh5"
Nov 03 00:00:00.435373 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.435109    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfxwl\" (UniqueName: \"kubernetes.io/projected/4ecf1fa4-ede5-420c-ace8-ef7bfb01423b-kube-api-access-zfxwl\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8\" (UID: \"4ecf1fa4-ede5-420c-ace8-ef7bfb01423b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8"
Nov 03 00:00:00.435373 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.435135    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbd6z\" (UniqueName: \"kubernetes.io/projected/50b27971-4ed6-44d2-972a-b5596ab2fffb-kube-api-access-jbd6z\") pod \"maas-api-key-cleanup-29894400-58nzv\" (UID: \"50b27971-4ed6-44d2-972a-b5596ab2fffb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv"
Nov 03 00:00:00.435373 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.435164    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e595d849-e5c1-49db-90f3-dc35a24158bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft\" (UID: \"e595d849-e5c1-49db-90f3-dc35a24158bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft"
Nov 03 00:00:00.435373 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.435183    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b27971-4ed6-44d2-972a-b5596ab2fffb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894400-58nzv\" (UID: \"50b27971-4ed6-44d2-972a-b5596ab2fffb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv"
Nov 03 00:00:00.435373 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.435252    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6xbb\" (UniqueName: \"kubernetes.io/projected/e595d849-e5c1-49db-90f3-dc35a24158bd-kube-api-access-s6xbb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft\" (UID: \"e595d849-e5c1-49db-90f3-dc35a24158bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft"
Nov 03 00:00:00.435373 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.435293    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ecf1fa4-ede5-420c-ace8-ef7bfb01423b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8\" (UID: \"4ecf1fa4-ede5-420c-ace8-ef7bfb01423b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8"
Nov 03 00:00:00.435373 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.435329    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b200fdb-64bb-40f0-9663-1d7f1e7e3723-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx\" (UID: \"8b200fdb-64bb-40f0-9663-1d7f1e7e3723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx"
Nov 03 00:00:00.435373 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.435358    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8c748df-cfe2-4382-b65c-dbf9937c03d9-serviceca\") pod \"image-pruner-29894400-vhdh5\" (UID: \"a8c748df-cfe2-4382-b65c-dbf9937c03d9\") " pod="openshift-image-registry/image-pruner-29894400-vhdh5"
Nov 03 00:00:00.437133 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.435945    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8c748df-cfe2-4382-b65c-dbf9937c03d9-serviceca\") pod \"image-pruner-29894400-vhdh5\" (UID: \"a8c748df-cfe2-4382-b65c-dbf9937c03d9\") " pod="openshift-image-registry/image-pruner-29894400-vhdh5"
Nov 03 00:00:00.437133 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.436419    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e595d849-e5c1-49db-90f3-dc35a24158bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft\" (UID: \"e595d849-e5c1-49db-90f3-dc35a24158bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft"
Nov 03 00:00:00.437133 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.436955    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c95f99-51cf-4420-bb11-d6022fd906f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht\" (UID: \"d9c95f99-51cf-4420-bb11-d6022fd906f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht"
Nov 03 00:00:00.437397 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.437184    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ecf1fa4-ede5-420c-ace8-ef7bfb01423b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8\" (UID: \"4ecf1fa4-ede5-420c-ace8-ef7bfb01423b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8"
Nov 03 00:00:00.437779 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.437754    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b200fdb-64bb-40f0-9663-1d7f1e7e3723-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx\" (UID: \"8b200fdb-64bb-40f0-9663-1d7f1e7e3723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx"
Nov 03 00:00:00.491598 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.491372    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dtgr\" (UniqueName: \"kubernetes.io/projected/8b200fdb-64bb-40f0-9663-1d7f1e7e3723-kube-api-access-5dtgr\") pod \"maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx\" (UID: \"8b200fdb-64bb-40f0-9663-1d7f1e7e3723\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx"
Nov 03 00:00:00.491598 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.491546    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6xbb\" (UniqueName: \"kubernetes.io/projected/e595d849-e5c1-49db-90f3-dc35a24158bd-kube-api-access-s6xbb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft\" (UID: \"e595d849-e5c1-49db-90f3-dc35a24158bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft"
Nov 03 00:00:00.493025 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.492858    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wxpb\" (UniqueName: \"kubernetes.io/projected/a8c748df-cfe2-4382-b65c-dbf9937c03d9-kube-api-access-6wxpb\") pod \"image-pruner-29894400-vhdh5\" (UID: \"a8c748df-cfe2-4382-b65c-dbf9937c03d9\") " pod="openshift-image-registry/image-pruner-29894400-vhdh5"
Nov 03 00:00:00.493025 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.492884    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzrlw\" (UniqueName: \"kubernetes.io/projected/d9c95f99-51cf-4420-bb11-d6022fd906f4-kube-api-access-wzrlw\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht\" (UID: \"d9c95f99-51cf-4420-bb11-d6022fd906f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht"
Nov 03 00:00:00.493971 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.493939    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfxwl\" (UniqueName: \"kubernetes.io/projected/4ecf1fa4-ede5-420c-ace8-ef7bfb01423b-kube-api-access-zfxwl\") pod \"maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8\" (UID: \"4ecf1fa4-ede5-420c-ace8-ef7bfb01423b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8"
Nov 03 00:00:00.521824 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.521785    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29894400-vhdh5"
Nov 03 00:00:00.533247 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.533213    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8"
Nov 03 00:00:00.536714 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.536683    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbd6z\" (UniqueName: \"kubernetes.io/projected/50b27971-4ed6-44d2-972a-b5596ab2fffb-kube-api-access-jbd6z\") pod \"maas-api-key-cleanup-29894400-58nzv\" (UID: \"50b27971-4ed6-44d2-972a-b5596ab2fffb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv"
Nov 03 00:00:00.536811 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.536750    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b27971-4ed6-44d2-972a-b5596ab2fffb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894400-58nzv\" (UID: \"50b27971-4ed6-44d2-972a-b5596ab2fffb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv"
Nov 03 00:00:00.537667 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.537640    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b27971-4ed6-44d2-972a-b5596ab2fffb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894400-58nzv\" (UID: \"50b27971-4ed6-44d2-972a-b5596ab2fffb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv"
Nov 03 00:00:00.545734 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.545538    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft"
Nov 03 00:00:00.570592 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.570558    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx"
Nov 03 00:00:00.578249 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.578220    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht"
Nov 03 00:00:00.602035 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.601940    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbd6z\" (UniqueName: \"kubernetes.io/projected/50b27971-4ed6-44d2-972a-b5596ab2fffb-kube-api-access-jbd6z\") pod \"maas-api-key-cleanup-29894400-58nzv\" (UID: \"50b27971-4ed6-44d2-972a-b5596ab2fffb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv"
Nov 03 00:00:00.632843 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.632535    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv"
Nov 03 00:00:00.931653 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:00.931624    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29894400-vhdh5"]
Nov 03 00:00:01.107389 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:01.107342    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:01.160049 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:01.160015    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv"]
Nov 03 00:00:01.160206 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:00:01.160095    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b27971_4ed6_44d2_972a_b5596ab2fffb.slice/crio-42c94d222eec1f61d8e626775d7a83a552668c872ca02d283344b6613b7c95ed WatchSource:0}: Error finding container 42c94d222eec1f61d8e626775d7a83a552668c872ca02d283344b6613b7c95ed: Status 404 returned error can't find the container with id 42c94d222eec1f61d8e626775d7a83a552668c872ca02d283344b6613b7c95ed
Nov 03 00:00:01.184705 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:01.182326    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8"]
Nov 03 00:00:01.191898 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:00:01.191711    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ecf1fa4_ede5_420c_ace8_ef7bfb01423b.slice/crio-98e042621e9ac623e804f88158d2b8507e15e2301686901109f8154975bd2faf WatchSource:0}: Error finding container 98e042621e9ac623e804f88158d2b8507e15e2301686901109f8154975bd2faf: Status 404 returned error can't find the container with id 98e042621e9ac623e804f88158d2b8507e15e2301686901109f8154975bd2faf
Nov 03 00:00:01.254710 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:00:01.254565    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode595d849_e5c1_49db_90f3_dc35a24158bd.slice/crio-aacde8366f5b383d175326c9f7d19ee58631e11f1e30a1fba0fb2b53b1538a4d WatchSource:0}: Error finding container aacde8366f5b383d175326c9f7d19ee58631e11f1e30a1fba0fb2b53b1538a4d: Status 404 returned error can't find the container with id aacde8366f5b383d175326c9f7d19ee58631e11f1e30a1fba0fb2b53b1538a4d
Nov 03 00:00:01.254710 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:01.254612    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft"]
Nov 03 00:00:01.255439 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:00:01.255402    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c95f99_51cf_4420_bb11_d6022fd906f4.slice/crio-cfc77f94096ee5637df88f045ff2174ab317f35693b5013500df0a0feab85635 WatchSource:0}: Error finding container cfc77f94096ee5637df88f045ff2174ab317f35693b5013500df0a0feab85635: Status 404 returned error can't find the container with id cfc77f94096ee5637df88f045ff2174ab317f35693b5013500df0a0feab85635
Nov 03 00:00:01.260224 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:01.260200    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht"]
Nov 03 00:00:01.302734 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:01.302702    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx"]
Nov 03 00:00:01.379653 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:01.379624    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx" event={"ID":"8b200fdb-64bb-40f0-9663-1d7f1e7e3723","Type":"ContainerStarted","Data":"696dacf0e6f0013f0a63b2b12d2ee33815243837589f83c98d8cd9bce103b82c"}
Nov 03 00:00:01.381547 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:01.381518    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft" event={"ID":"e595d849-e5c1-49db-90f3-dc35a24158bd","Type":"ContainerStarted","Data":"aacde8366f5b383d175326c9f7d19ee58631e11f1e30a1fba0fb2b53b1538a4d"}
Nov 03 00:00:01.385042 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:01.385010    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8" event={"ID":"4ecf1fa4-ede5-420c-ace8-ef7bfb01423b","Type":"ContainerStarted","Data":"98e042621e9ac623e804f88158d2b8507e15e2301686901109f8154975bd2faf"}
Nov 03 00:00:01.390156 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:01.390121    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29894400-vhdh5" event={"ID":"a8c748df-cfe2-4382-b65c-dbf9937c03d9","Type":"ContainerStarted","Data":"352bcba835f845867c4fb817ffe07f9b2c3e912f429e2930264e6148f394c8ed"}
Nov 03 00:00:01.390261 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:01.390181    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29894400-vhdh5" event={"ID":"a8c748df-cfe2-4382-b65c-dbf9937c03d9","Type":"ContainerStarted","Data":"50cdfa5604e18f9c4bd5d0371ebfd4c1b7ef4f11134201858ee57cefe06ea562"}
Nov 03 00:00:01.393371 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:01.393053    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv" event={"ID":"50b27971-4ed6-44d2-972a-b5596ab2fffb","Type":"ContainerStarted","Data":"42c94d222eec1f61d8e626775d7a83a552668c872ca02d283344b6613b7c95ed"}
Nov 03 00:00:01.394815 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:01.394751    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht" event={"ID":"d9c95f99-51cf-4420-bb11-d6022fd906f4","Type":"ContainerStarted","Data":"cfc77f94096ee5637df88f045ff2174ab317f35693b5013500df0a0feab85635"}
Nov 03 00:00:01.504498 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:01.504423    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-pruner-29894400-vhdh5" podStartSLOduration=1.504403851 podStartE2EDuration="1.504403851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:01.498646652 +0000 UTC m=+4041.067001441" watchObservedRunningTime="2026-11-03 00:00:01.504403851 +0000 UTC m=+4041.072758641"
Nov 03 00:00:02.312439 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:02.312066    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj"
Nov 03 00:00:02.312439 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:02.312159    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6"
Nov 03 00:00:02.415907 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:02.414992    4294 generic.go:356] "Generic (PLEG): container finished" podID="4ecf1fa4-ede5-420c-ace8-ef7bfb01423b" containerID="1ebe617a50345bfa2f5f19a6dba2e396fc20d070b0f5a9dcffc8681f12f39736" exitCode=60
Nov 03 00:00:02.415907 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:02.415159    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8" event={"ID":"4ecf1fa4-ede5-420c-ace8-ef7bfb01423b","Type":"ContainerDied","Data":"1ebe617a50345bfa2f5f19a6dba2e396fc20d070b0f5a9dcffc8681f12f39736"}
Nov 03 00:00:02.415907 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:02.415555    4294 scope.go:122] "RemoveContainer" containerID="1ebe617a50345bfa2f5f19a6dba2e396fc20d070b0f5a9dcffc8681f12f39736"
Nov 03 00:00:02.426022 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:02.425113    4294 generic.go:356] "Generic (PLEG): container finished" podID="50b27971-4ed6-44d2-972a-b5596ab2fffb" containerID="bf473c4bc409d8a5d9b7ff9ea58fc39eeb849c090944e820c20f20c67405a6fb" exitCode=60
Nov 03 00:00:02.426022 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:02.425200    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv" event={"ID":"50b27971-4ed6-44d2-972a-b5596ab2fffb","Type":"ContainerDied","Data":"bf473c4bc409d8a5d9b7ff9ea58fc39eeb849c090944e820c20f20c67405a6fb"}
Nov 03 00:00:02.426022 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:02.425602    4294 scope.go:122] "RemoveContainer" containerID="bf473c4bc409d8a5d9b7ff9ea58fc39eeb849c090944e820c20f20c67405a6fb"
Nov 03 00:00:02.429948 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:02.429510    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht" event={"ID":"d9c95f99-51cf-4420-bb11-d6022fd906f4","Type":"ContainerStarted","Data":"6acce0b0401236446ac572746b54e5526d5425c89072d55cea6eb1c8ee9f0561"}
Nov 03 00:00:02.431860 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:02.431837    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx" event={"ID":"8b200fdb-64bb-40f0-9663-1d7f1e7e3723","Type":"ContainerStarted","Data":"3a57d50e308f030b28f56806dc7e0a8a0a556941293e7981bdf9d87d4cd2bbfa"}
Nov 03 00:00:02.435428 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:02.435402    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft" event={"ID":"e595d849-e5c1-49db-90f3-dc35a24158bd","Type":"ContainerStarted","Data":"b7d171383681d21195c325347d3c0742fd299a901102609c809eb7359d9adbf3"}
Nov 03 00:00:02.804346 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:02.795330    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx" podStartSLOduration=2.795310777 podStartE2EDuration="2.795310777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:02.789303914 +0000 UTC m=+4042.357658703" watchObservedRunningTime="2026-11-03 00:00:02.795310777 +0000 UTC m=+4042.363665568"
Nov 03 00:00:03.007503 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:03.007443    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft" podStartSLOduration=3.007427345 podStartE2EDuration="3.007427345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:03.005878705 +0000 UTC m=+4042.574233540" watchObservedRunningTime="2026-11-03 00:00:03.007427345 +0000 UTC m=+4042.575782131"
Nov 03 00:00:03.118029 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:03.117900    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht" podStartSLOduration=3.117881137 podStartE2EDuration="3.117881137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:03.112659964 +0000 UTC m=+4042.681014765" watchObservedRunningTime="2026-11-03 00:00:03.117881137 +0000 UTC m=+4042.686235925"
Nov 03 00:00:03.241961 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:03.241922    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:00:03.444131 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:03.444099    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv" event={"ID":"50b27971-4ed6-44d2-972a-b5596ab2fffb","Type":"ContainerStarted","Data":"6df65e14caee1c3d84f109ba44d39a1b6356162f234fa650216d485636709921"}
Nov 03 00:00:03.446267 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:03.446244    4294 generic.go:356] "Generic (PLEG): container finished" podID="d9c95f99-51cf-4420-bb11-d6022fd906f4" containerID="6acce0b0401236446ac572746b54e5526d5425c89072d55cea6eb1c8ee9f0561" exitCode=60
Nov 03 00:00:03.446401 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:03.446327    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht" event={"ID":"d9c95f99-51cf-4420-bb11-d6022fd906f4","Type":"ContainerDied","Data":"6acce0b0401236446ac572746b54e5526d5425c89072d55cea6eb1c8ee9f0561"}
Nov 03 00:00:03.446567 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:03.446551    4294 scope.go:122] "RemoveContainer" containerID="6acce0b0401236446ac572746b54e5526d5425c89072d55cea6eb1c8ee9f0561"
Nov 03 00:00:03.448164 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:03.448136    4294 generic.go:356] "Generic (PLEG): container finished" podID="8b200fdb-64bb-40f0-9663-1d7f1e7e3723" containerID="3a57d50e308f030b28f56806dc7e0a8a0a556941293e7981bdf9d87d4cd2bbfa" exitCode=60
Nov 03 00:00:03.448270 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:03.448168    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx" event={"ID":"8b200fdb-64bb-40f0-9663-1d7f1e7e3723","Type":"ContainerDied","Data":"3a57d50e308f030b28f56806dc7e0a8a0a556941293e7981bdf9d87d4cd2bbfa"}
Nov 03 00:00:03.448453 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:03.448437    4294 scope.go:122] "RemoveContainer" containerID="3a57d50e308f030b28f56806dc7e0a8a0a556941293e7981bdf9d87d4cd2bbfa"
Nov 03 00:00:03.450343 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:03.450116    4294 generic.go:356] "Generic (PLEG): container finished" podID="e595d849-e5c1-49db-90f3-dc35a24158bd" containerID="b7d171383681d21195c325347d3c0742fd299a901102609c809eb7359d9adbf3" exitCode=60
Nov 03 00:00:03.450343 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:03.450201    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft" event={"ID":"e595d849-e5c1-49db-90f3-dc35a24158bd","Type":"ContainerDied","Data":"b7d171383681d21195c325347d3c0742fd299a901102609c809eb7359d9adbf3"}
Nov 03 00:00:03.450477 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:03.450372    4294 scope.go:122] "RemoveContainer" containerID="b7d171383681d21195c325347d3c0742fd299a901102609c809eb7359d9adbf3"
Nov 03 00:00:03.452864 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:03.452844    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8" event={"ID":"4ecf1fa4-ede5-420c-ace8-ef7bfb01423b","Type":"ContainerStarted","Data":"9dd066d75164a6d2a0784a003a2675894ce3db41b9ca4ecedd3eda2f0438aa4e"}
Nov 03 00:00:03.555511 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:03.555442    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv" podStartSLOduration=3.555421288 podStartE2EDuration="3.555421288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:03.552668419 +0000 UTC m=+4043.121023206" watchObservedRunningTime="2026-11-03 00:00:03.555421288 +0000 UTC m=+4043.123776082"
Nov 03 00:00:03.990864 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:03.990816    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8" podStartSLOduration=3.990802093 podStartE2EDuration="3.990802093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:03.987458794 +0000 UTC m=+4043.555813588" watchObservedRunningTime="2026-11-03 00:00:03.990802093 +0000 UTC m=+4043.559156881"
Nov 03 00:00:04.462392 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:04.462363    4294 generic.go:356] "Generic (PLEG): container finished" podID="4ecf1fa4-ede5-420c-ace8-ef7bfb01423b" containerID="9dd066d75164a6d2a0784a003a2675894ce3db41b9ca4ecedd3eda2f0438aa4e" exitCode=60
Nov 03 00:00:04.462745 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:04.462441    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8" event={"ID":"4ecf1fa4-ede5-420c-ace8-ef7bfb01423b","Type":"ContainerDied","Data":"9dd066d75164a6d2a0784a003a2675894ce3db41b9ca4ecedd3eda2f0438aa4e"}
Nov 03 00:00:04.462745 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:04.462502    4294 scope.go:122] "RemoveContainer" containerID="1ebe617a50345bfa2f5f19a6dba2e396fc20d070b0f5a9dcffc8681f12f39736"
Nov 03 00:00:04.462745 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:04.462709    4294 scope.go:122] "RemoveContainer" containerID="9dd066d75164a6d2a0784a003a2675894ce3db41b9ca4ecedd3eda2f0438aa4e"
Nov 03 00:00:04.463167 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:04.463134    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8_odh-ai-gateway-infra(4ecf1fa4-ede5-420c-ace8-ef7bfb01423b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8" podUID="4ecf1fa4-ede5-420c-ace8-ef7bfb01423b"
Nov 03 00:00:04.464179 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:04.464159    4294 generic.go:356] "Generic (PLEG): container finished" podID="a8c748df-cfe2-4382-b65c-dbf9937c03d9" containerID="352bcba835f845867c4fb817ffe07f9b2c3e912f429e2930264e6148f394c8ed" exitCode=0
Nov 03 00:00:04.464307 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:04.464242    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29894400-vhdh5" event={"ID":"a8c748df-cfe2-4382-b65c-dbf9937c03d9","Type":"ContainerDied","Data":"352bcba835f845867c4fb817ffe07f9b2c3e912f429e2930264e6148f394c8ed"}
Nov 03 00:00:04.466014 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:04.465972    4294 generic.go:356] "Generic (PLEG): container finished" podID="50b27971-4ed6-44d2-972a-b5596ab2fffb" containerID="6df65e14caee1c3d84f109ba44d39a1b6356162f234fa650216d485636709921" exitCode=60
Nov 03 00:00:04.466093 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:04.466028    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv" event={"ID":"50b27971-4ed6-44d2-972a-b5596ab2fffb","Type":"ContainerDied","Data":"6df65e14caee1c3d84f109ba44d39a1b6356162f234fa650216d485636709921"}
Nov 03 00:00:04.466292 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:04.466279    4294 scope.go:122] "RemoveContainer" containerID="6df65e14caee1c3d84f109ba44d39a1b6356162f234fa650216d485636709921"
Nov 03 00:00:04.466609 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:04.466584    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894400-58nzv_odh-ai-gateway-infra(50b27971-4ed6-44d2-972a-b5596ab2fffb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv" podUID="50b27971-4ed6-44d2-972a-b5596ab2fffb"
Nov 03 00:00:04.467909 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:04.467885    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht" event={"ID":"d9c95f99-51cf-4420-bb11-d6022fd906f4","Type":"ContainerStarted","Data":"033e342982292681f9ee237654d54d40a22043b24b75a53bb76a517f8df743ba"}
Nov 03 00:00:04.468145 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:04.468130    4294 scope.go:122] "RemoveContainer" containerID="033e342982292681f9ee237654d54d40a22043b24b75a53bb76a517f8df743ba"
Nov 03 00:00:04.468576 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:04.468345    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht_odh-ai-gateway-infra(d9c95f99-51cf-4420-bb11-d6022fd906f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht" podUID="d9c95f99-51cf-4420-bb11-d6022fd906f4"
Nov 03 00:00:04.469808 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:04.469790    4294 generic.go:356] "Generic (PLEG): container finished" podID="8b200fdb-64bb-40f0-9663-1d7f1e7e3723" containerID="a451e7ffab922be199a83ed64aa22147d6df312a53f23dd08b22e384958e080e" exitCode=60
Nov 03 00:00:04.469902 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:04.469854    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx" event={"ID":"8b200fdb-64bb-40f0-9663-1d7f1e7e3723","Type":"ContainerDied","Data":"a451e7ffab922be199a83ed64aa22147d6df312a53f23dd08b22e384958e080e"}
Nov 03 00:00:04.470088 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:04.470072    4294 scope.go:122] "RemoveContainer" containerID="a451e7ffab922be199a83ed64aa22147d6df312a53f23dd08b22e384958e080e"
Nov 03 00:00:04.470328 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:04.470309    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx_odh-ai-gateway-infra(8b200fdb-64bb-40f0-9663-1d7f1e7e3723)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx" podUID="8b200fdb-64bb-40f0-9663-1d7f1e7e3723"
Nov 03 00:00:04.471590 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:04.471572    4294 generic.go:356] "Generic (PLEG): container finished" podID="e595d849-e5c1-49db-90f3-dc35a24158bd" containerID="efe29c878dfa5dc0b2672e9f48bdb64c346744655e38097786567b0d24c11412" exitCode=60
Nov 03 00:00:04.471670 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:04.471639    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft" event={"ID":"e595d849-e5c1-49db-90f3-dc35a24158bd","Type":"ContainerDied","Data":"efe29c878dfa5dc0b2672e9f48bdb64c346744655e38097786567b0d24c11412"}
Nov 03 00:00:04.471851 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:04.471837    4294 scope.go:122] "RemoveContainer" containerID="efe29c878dfa5dc0b2672e9f48bdb64c346744655e38097786567b0d24c11412"
Nov 03 00:00:04.472088 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:04.472066    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft_odh-ai-gateway-infra(e595d849-e5c1-49db-90f3-dc35a24158bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft" podUID="e595d849-e5c1-49db-90f3-dc35a24158bd"
Nov 03 00:00:04.551711 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:04.551691    4294 scope.go:122] "RemoveContainer" containerID="bf473c4bc409d8a5d9b7ff9ea58fc39eeb849c090944e820c20f20c67405a6fb"
Nov 03 00:00:04.565701 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:04.565684    4294 scope.go:122] "RemoveContainer" containerID="3a57d50e308f030b28f56806dc7e0a8a0a556941293e7981bdf9d87d4cd2bbfa"
Nov 03 00:00:04.579733 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:04.579711    4294 scope.go:122] "RemoveContainer" containerID="b7d171383681d21195c325347d3c0742fd299a901102609c809eb7359d9adbf3"
Nov 03 00:00:05.479942 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:05.479914    4294 scope.go:122] "RemoveContainer" containerID="efe29c878dfa5dc0b2672e9f48bdb64c346744655e38097786567b0d24c11412"
Nov 03 00:00:05.480421 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:05.480262    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft_odh-ai-gateway-infra(e595d849-e5c1-49db-90f3-dc35a24158bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft" podUID="e595d849-e5c1-49db-90f3-dc35a24158bd"
Nov 03 00:00:05.481532 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:05.481507    4294 scope.go:122] "RemoveContainer" containerID="9dd066d75164a6d2a0784a003a2675894ce3db41b9ca4ecedd3eda2f0438aa4e"
Nov 03 00:00:05.481781 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:05.481761    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8_odh-ai-gateway-infra(4ecf1fa4-ede5-420c-ace8-ef7bfb01423b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8" podUID="4ecf1fa4-ede5-420c-ace8-ef7bfb01423b"
Nov 03 00:00:05.483024 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:05.483005    4294 scope.go:122] "RemoveContainer" containerID="6df65e14caee1c3d84f109ba44d39a1b6356162f234fa650216d485636709921"
Nov 03 00:00:05.483240 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:05.483213    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894400-58nzv_odh-ai-gateway-infra(50b27971-4ed6-44d2-972a-b5596ab2fffb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv" podUID="50b27971-4ed6-44d2-972a-b5596ab2fffb"
Nov 03 00:00:05.484605 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:05.484586    4294 generic.go:356] "Generic (PLEG): container finished" podID="d9c95f99-51cf-4420-bb11-d6022fd906f4" containerID="033e342982292681f9ee237654d54d40a22043b24b75a53bb76a517f8df743ba" exitCode=60
Nov 03 00:00:05.484689 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:05.484638    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht" event={"ID":"d9c95f99-51cf-4420-bb11-d6022fd906f4","Type":"ContainerDied","Data":"033e342982292681f9ee237654d54d40a22043b24b75a53bb76a517f8df743ba"}
Nov 03 00:00:05.484733 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:05.484692    4294 scope.go:122] "RemoveContainer" containerID="6acce0b0401236446ac572746b54e5526d5425c89072d55cea6eb1c8ee9f0561"
Nov 03 00:00:05.484883 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:05.484865    4294 scope.go:122] "RemoveContainer" containerID="033e342982292681f9ee237654d54d40a22043b24b75a53bb76a517f8df743ba"
Nov 03 00:00:05.485156 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:05.485122    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht_odh-ai-gateway-infra(d9c95f99-51cf-4420-bb11-d6022fd906f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht" podUID="d9c95f99-51cf-4420-bb11-d6022fd906f4"
Nov 03 00:00:05.486680 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:05.486626    4294 scope.go:122] "RemoveContainer" containerID="a451e7ffab922be199a83ed64aa22147d6df312a53f23dd08b22e384958e080e"
Nov 03 00:00:05.486823 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:05.486805    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx_odh-ai-gateway-infra(8b200fdb-64bb-40f0-9663-1d7f1e7e3723)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx" podUID="8b200fdb-64bb-40f0-9663-1d7f1e7e3723"
Nov 03 00:00:05.635514 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:05.635488    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29894400-vhdh5"
Nov 03 00:00:05.663735 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:05.663708    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c748df-cfe2-4382-b65c-dbf9937c03d9-serviceca\" (UniqueName: \"kubernetes.io/configmap/a8c748df-cfe2-4382-b65c-dbf9937c03d9-serviceca\") pod \"a8c748df-cfe2-4382-b65c-dbf9937c03d9\" (UID: \"a8c748df-cfe2-4382-b65c-dbf9937c03d9\") "
Nov 03 00:00:05.663880 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:05.663862    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c748df-cfe2-4382-b65c-dbf9937c03d9-kube-api-access-6wxpb\" (UniqueName: \"kubernetes.io/projected/a8c748df-cfe2-4382-b65c-dbf9937c03d9-kube-api-access-6wxpb\") pod \"a8c748df-cfe2-4382-b65c-dbf9937c03d9\" (UID: \"a8c748df-cfe2-4382-b65c-dbf9937c03d9\") "
Nov 03 00:00:05.664046 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:05.664027    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c748df-cfe2-4382-b65c-dbf9937c03d9-serviceca" pod "a8c748df-cfe2-4382-b65c-dbf9937c03d9" (UID: "a8c748df-cfe2-4382-b65c-dbf9937c03d9"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:00:05.664289 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:05.664274    4294 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a8c748df-cfe2-4382-b65c-dbf9937c03d9-serviceca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:00:05.665921 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:05.665900    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c748df-cfe2-4382-b65c-dbf9937c03d9-kube-api-access-6wxpb" pod "a8c748df-cfe2-4382-b65c-dbf9937c03d9" (UID: "a8c748df-cfe2-4382-b65c-dbf9937c03d9"). InnerVolumeSpecName "kube-api-access-6wxpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:00:05.765518 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:05.765490    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wxpb\" (UniqueName: \"kubernetes.io/projected/a8c748df-cfe2-4382-b65c-dbf9937c03d9-kube-api-access-6wxpb\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:00:05.935507 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:05.935469    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:06.493824 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:06.493797    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29894400-vhdh5"
Nov 03 00:00:06.494307 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:06.493829    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29894400-vhdh5" event={"ID":"a8c748df-cfe2-4382-b65c-dbf9937c03d9","Type":"ContainerDied","Data":"50cdfa5604e18f9c4bd5d0371ebfd4c1b7ef4f11134201858ee57cefe06ea562"}
Nov 03 00:00:06.494307 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:06.493874    4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50cdfa5604e18f9c4bd5d0371ebfd4c1b7ef4f11134201858ee57cefe06ea562"
Nov 03 00:00:06.495820 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:06.495802    4294 scope.go:122] "RemoveContainer" containerID="033e342982292681f9ee237654d54d40a22043b24b75a53bb76a517f8df743ba"
Nov 03 00:00:06.496058 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:06.496038    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht_odh-ai-gateway-infra(d9c95f99-51cf-4420-bb11-d6022fd906f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht" podUID="d9c95f99-51cf-4420-bb11-d6022fd906f4"
Nov 03 00:00:09.298023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.297990    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"]
Nov 03 00:00:09.298690 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.298676    4294 cpu_manager.go:414] "containerMap: removing container" podUID="a8c748df-cfe2-4382-b65c-dbf9937c03d9" containerName="image-pruner"
Nov 03 00:00:09.298740 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.298691    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c748df-cfe2-4382-b65c-dbf9937c03d9" containerName="image-pruner"
Nov 03 00:00:09.298805 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.298796    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c748df-cfe2-4382-b65c-dbf9937c03d9" containerName="image-pruner"
Nov 03 00:00:09.302488 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.302472    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.345766 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.345739    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"]
Nov 03 00:00:09.350078 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.350062    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.396146 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.396112    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"]
Nov 03 00:00:09.409172 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.409131    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c23a4b6-9cde-465d-9539-41c313ccc802-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.409357 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.409208    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx9fj\" (UniqueName: \"kubernetes.io/projected/9c23a4b6-9cde-465d-9539-41c313ccc802-kube-api-access-gx9fj\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.409357 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.409304    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.409479 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.409361    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.409479 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.409413    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.409479 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.409460    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.409583 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.409498    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m6tb\" (UniqueName: \"kubernetes.io/projected/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-kube-api-access-4m6tb\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.409583 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.409548    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.409657 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.409598    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.409657 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.409623    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.409723 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.409685    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.409723 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.409714    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.409794 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.409743    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.409794 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.409767    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.409854 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.409805    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.409888 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.409849    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.409924 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.409902    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.409959 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.409942    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.442879 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.442849    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"]
Nov 03 00:00:09.512032 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.511961    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c23a4b6-9cde-465d-9539-41c313ccc802-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.512274 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.512071    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx9fj\" (UniqueName: \"kubernetes.io/projected/9c23a4b6-9cde-465d-9539-41c313ccc802-kube-api-access-gx9fj\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.512274 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.512133    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.512274 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.512159    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.512274 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.512197    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.512274 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.512248    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.512616 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.512286    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m6tb\" (UniqueName: \"kubernetes.io/projected/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-kube-api-access-4m6tb\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.512616 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.512339    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.512616 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.512393    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.512616 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.512420    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.512616 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.512486    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.512616 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.512514    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.512616 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.512548    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.512616 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.512571    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.512616 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.512609    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.513117 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.512656    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.513117 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.512713    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.513117 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.512755    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.513117 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.512902    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c23a4b6-9cde-465d-9539-41c313ccc802-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.513334 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.513263    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.513655 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.513627    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.513793 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.513763    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.513930 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.513913    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.514098 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.514073    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.514360 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.514335    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.514641 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.514615    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.514893 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.514869    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.515029 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.515004    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.516508 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.516480    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.516633 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.516611    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.516695 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.516610    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.516695 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.516682    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.567600 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.567505    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.567600 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.567577    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx9fj\" (UniqueName: \"kubernetes.io/projected/9c23a4b6-9cde-465d-9539-41c313ccc802-kube-api-access-gx9fj\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-xgcsj\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:09.567832 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.567690    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.568565 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.568539    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m6tb\" (UniqueName: \"kubernetes.io/projected/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-kube-api-access-4m6tb\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-cvprl\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.616672 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.616639    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:09.663276 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:09.663241    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:10.032698 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:00:10.032672    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9dbf5e_0447_4975_b1f3_e74e374ee7a2.slice/crio-26ea640f77a5687010755686c2d44da587e43929c21292221f2dbccd999133a3 WatchSource:0}: Error finding container 26ea640f77a5687010755686c2d44da587e43929c21292221f2dbccd999133a3: Status 404 returned error can't find the container with id 26ea640f77a5687010755686c2d44da587e43929c21292221f2dbccd999133a3
Nov 03 00:00:10.033556 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:10.033532    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"]
Nov 03 00:00:10.035327 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:10.035295    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:00:10.035410 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:10.035369    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:00:10.035410 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:10.035406    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:00:10.080787 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:00:10.080764    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c23a4b6_9cde_465d_9539_41c313ccc802.slice/crio-4bdb0119bded295c310158ca4ee2f23e29737a218f970f512e93b7b02add315b WatchSource:0}: Error finding container 4bdb0119bded295c310158ca4ee2f23e29737a218f970f512e93b7b02add315b: Status 404 returned error can't find the container with id 4bdb0119bded295c310158ca4ee2f23e29737a218f970f512e93b7b02add315b
Nov 03 00:00:10.081924 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:10.081903    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"]
Nov 03 00:00:10.083541 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:10.083514    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:00:10.083617 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:10.083586    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:00:10.083656 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:10.083612    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:00:10.419308 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:10.419186    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:10.531490 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:10.531448    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj" event={"ID":"9c23a4b6-9cde-465d-9539-41c313ccc802","Type":"ContainerStarted","Data":"e98478fade349d6cc1f39660277781690dbc1193c80a7967e6a3a0a9d6d0353a"}
Nov 03 00:00:10.531665 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:10.531519    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj" event={"ID":"9c23a4b6-9cde-465d-9539-41c313ccc802","Type":"ContainerStarted","Data":"4bdb0119bded295c310158ca4ee2f23e29737a218f970f512e93b7b02add315b"}
Nov 03 00:00:10.533621 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:10.533592    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl" event={"ID":"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2","Type":"ContainerStarted","Data":"9c0fd3178fe34d6578050fcd34a0e3df3be7be9504e330de18390358fc03f96e"}
Nov 03 00:00:10.533762 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:10.533646    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl" event={"ID":"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2","Type":"ContainerStarted","Data":"26ea640f77a5687010755686c2d44da587e43929c21292221f2dbccd999133a3"}
Nov 03 00:00:10.617527 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:10.617497    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:10.644566 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:10.644504    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj" podStartSLOduration=1.644486352 podStartE2EDuration="1.644486352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:10.642518404 +0000 UTC m=+4050.210873191" watchObservedRunningTime="2026-11-03 00:00:10.644486352 +0000 UTC m=+4050.212841140"
Nov 03 00:00:10.663545 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:10.663519    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:10.754804 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:10.754749    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl" podStartSLOduration=1.754737166 podStartE2EDuration="1.754737166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:10.754029192 +0000 UTC m=+4050.322383979" watchObservedRunningTime="2026-11-03 00:00:10.754737166 +0000 UTC m=+4050.323091953"
Nov 03 00:00:11.619410 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:11.619369    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cvprl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 00:00:11.619814 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:11.619425    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl" podUID="1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 00:00:11.672007 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:11.668023    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 00:00:11.672007 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:11.668089    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj" podUID="9c23a4b6-9cde-465d-9539-41c313ccc802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 00:00:11.793004 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:11.792905    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:00:12.617534 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:12.617489    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cvprl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 00:00:12.617727 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:12.617553    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl" podUID="1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 00:00:12.667004 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:12.666411    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 00:00:12.667004 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:12.666474    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj" podUID="9c23a4b6-9cde-465d-9539-41c313ccc802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 00:00:13.242017 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:13.241963    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:00:13.617170 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:13.617088    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cvprl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 00:00:13.617170 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:13.617143    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl" podUID="1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 00:00:13.676013 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:13.675359    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 00:00:13.676013 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:13.675419    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj" podUID="9c23a4b6-9cde-465d-9539-41c313ccc802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 00:00:14.541188 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:14.541154    4294 scope.go:122] "RemoveContainer" containerID="9dd066d75164a6d2a0784a003a2675894ce3db41b9ca4ecedd3eda2f0438aa4e"
Nov 03 00:00:14.617866 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:14.617837    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cvprl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 00:00:14.618049 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:14.618031    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl" podUID="1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 00:00:14.666506 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:14.666375    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 00:00:14.666506 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:14.666443    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj" podUID="9c23a4b6-9cde-465d-9539-41c313ccc802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 00:00:15.012247 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:15.012219    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:15.616972 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:15.616941    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cvprl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 00:00:15.617152 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:15.617009    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl" podUID="1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 00:00:15.617742 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:15.617723    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8" event={"ID":"4ecf1fa4-ede5-420c-ace8-ef7bfb01423b","Type":"ContainerStarted","Data":"3db131b4abb5d8fee74ec4fb6b72bb70d91dc6afbf6e6d5e880974427917d16c"}
Nov 03 00:00:15.665306 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:15.665274    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 00:00:15.665533 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:15.665508    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj" podUID="9c23a4b6-9cde-465d-9539-41c313ccc802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 00:00:16.540596 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:16.540562    4294 scope.go:122] "RemoveContainer" containerID="6df65e14caee1c3d84f109ba44d39a1b6356162f234fa650216d485636709921"
Nov 03 00:00:16.617119 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:16.617083    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cvprl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 00:00:16.617245 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:16.617167    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl" podUID="1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 00:00:16.628016 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:16.627989    4294 generic.go:356] "Generic (PLEG): container finished" podID="4ecf1fa4-ede5-420c-ace8-ef7bfb01423b" containerID="3db131b4abb5d8fee74ec4fb6b72bb70d91dc6afbf6e6d5e880974427917d16c" exitCode=60
Nov 03 00:00:16.628137 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:16.628046    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8" event={"ID":"4ecf1fa4-ede5-420c-ace8-ef7bfb01423b","Type":"ContainerDied","Data":"3db131b4abb5d8fee74ec4fb6b72bb70d91dc6afbf6e6d5e880974427917d16c"}
Nov 03 00:00:16.628137 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:16.628094    4294 scope.go:122] "RemoveContainer" containerID="9dd066d75164a6d2a0784a003a2675894ce3db41b9ca4ecedd3eda2f0438aa4e"
Nov 03 00:00:16.628503 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:16.628483    4294 scope.go:122] "RemoveContainer" containerID="3db131b4abb5d8fee74ec4fb6b72bb70d91dc6afbf6e6d5e880974427917d16c"
Nov 03 00:00:16.628859 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:16.628839    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8_odh-ai-gateway-infra(4ecf1fa4-ede5-420c-ace8-ef7bfb01423b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8" podUID="4ecf1fa4-ede5-420c-ace8-ef7bfb01423b"
Nov 03 00:00:16.667409 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:16.667373    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 00:00:16.667653 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:16.667623    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj" podUID="9c23a4b6-9cde-465d-9539-41c313ccc802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 00:00:16.751994 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:16.751943    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8"]
Nov 03 00:00:17.541385 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:17.541351    4294 scope.go:122] "RemoveContainer" containerID="efe29c878dfa5dc0b2672e9f48bdb64c346744655e38097786567b0d24c11412"
Nov 03 00:00:17.617538 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:17.617509    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cvprl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 00:00:17.617665 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:17.617567    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl" podUID="1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 00:00:17.641967 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:17.641942    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv" event={"ID":"50b27971-4ed6-44d2-972a-b5596ab2fffb","Type":"ContainerStarted","Data":"5d695f78e4457913902b73cb56e74e966fbf0475c68d8a6c2108659fb7ec9ca8"}
Nov 03 00:00:17.663885 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:17.663852    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 00:00:17.664036 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:17.663909    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj" podUID="9c23a4b6-9cde-465d-9539-41c313ccc802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 00:00:17.814324 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:17.814301    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8"
Nov 03 00:00:17.970763 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:17.970686    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ecf1fa4-ede5-420c-ace8-ef7bfb01423b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ecf1fa4-ede5-420c-ace8-ef7bfb01423b-maas-api-service-ca\") pod \"4ecf1fa4-ede5-420c-ace8-ef7bfb01423b\" (UID: \"4ecf1fa4-ede5-420c-ace8-ef7bfb01423b\") "
Nov 03 00:00:17.970935 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:17.970872    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ecf1fa4-ede5-420c-ace8-ef7bfb01423b-kube-api-access-zfxwl\" (UniqueName: \"kubernetes.io/projected/4ecf1fa4-ede5-420c-ace8-ef7bfb01423b-kube-api-access-zfxwl\") pod \"4ecf1fa4-ede5-420c-ace8-ef7bfb01423b\" (UID: \"4ecf1fa4-ede5-420c-ace8-ef7bfb01423b\") "
Nov 03 00:00:17.983005 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:17.971913    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ecf1fa4-ede5-420c-ace8-ef7bfb01423b-maas-api-service-ca" pod "4ecf1fa4-ede5-420c-ace8-ef7bfb01423b" (UID: "4ecf1fa4-ede5-420c-ace8-ef7bfb01423b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:00:17.983005 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:17.982904    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ecf1fa4-ede5-420c-ace8-ef7bfb01423b-kube-api-access-zfxwl" pod "4ecf1fa4-ede5-420c-ace8-ef7bfb01423b" (UID: "4ecf1fa4-ede5-420c-ace8-ef7bfb01423b"). InnerVolumeSpecName "kube-api-access-zfxwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:00:18.073613 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:18.073534    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfxwl\" (UniqueName: \"kubernetes.io/projected/4ecf1fa4-ede5-420c-ace8-ef7bfb01423b-kube-api-access-zfxwl\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:00:18.073613 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:18.073570    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ecf1fa4-ede5-420c-ace8-ef7bfb01423b-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:00:18.540706 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:18.540673    4294 scope.go:122] "RemoveContainer" containerID="a451e7ffab922be199a83ed64aa22147d6df312a53f23dd08b22e384958e080e"
Nov 03 00:00:18.540885 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:18.540775    4294 scope.go:122] "RemoveContainer" containerID="033e342982292681f9ee237654d54d40a22043b24b75a53bb76a517f8df743ba"
Nov 03 00:00:18.657012 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:18.654663    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8" event={"ID":"4ecf1fa4-ede5-420c-ace8-ef7bfb01423b","Type":"ContainerDied","Data":"98e042621e9ac623e804f88158d2b8507e15e2301686901109f8154975bd2faf"}
Nov 03 00:00:18.657012 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:18.654739    4294 scope.go:122] "RemoveContainer" containerID="3db131b4abb5d8fee74ec4fb6b72bb70d91dc6afbf6e6d5e880974427917d16c"
Nov 03 00:00:18.657012 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:18.654845    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8"
Nov 03 00:00:18.662275 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:18.662247    4294 generic.go:356] "Generic (PLEG): container finished" podID="50b27971-4ed6-44d2-972a-b5596ab2fffb" containerID="5d695f78e4457913902b73cb56e74e966fbf0475c68d8a6c2108659fb7ec9ca8" exitCode=60
Nov 03 00:00:18.662563 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:18.662540    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv" event={"ID":"50b27971-4ed6-44d2-972a-b5596ab2fffb","Type":"ContainerDied","Data":"5d695f78e4457913902b73cb56e74e966fbf0475c68d8a6c2108659fb7ec9ca8"}
Nov 03 00:00:18.664670 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:18.664649    4294 scope.go:122] "RemoveContainer" containerID="5d695f78e4457913902b73cb56e74e966fbf0475c68d8a6c2108659fb7ec9ca8"
Nov 03 00:00:18.665183 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:18.665158    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894400-58nzv_odh-ai-gateway-infra(50b27971-4ed6-44d2-972a-b5596ab2fffb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv" podUID="50b27971-4ed6-44d2-972a-b5596ab2fffb"
Nov 03 00:00:18.666321 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:18.666295    4294 generic.go:356] "Generic (PLEG): container finished" podID="e595d849-e5c1-49db-90f3-dc35a24158bd" containerID="832c6a23342dd2effa291183aad3dc0394ced2ab8f2da962c33029590e9eb2b1" exitCode=60
Nov 03 00:00:18.666413 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:18.666391    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft" event={"ID":"e595d849-e5c1-49db-90f3-dc35a24158bd","Type":"ContainerDied","Data":"832c6a23342dd2effa291183aad3dc0394ced2ab8f2da962c33029590e9eb2b1"}
Nov 03 00:00:18.666808 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:18.666780    4294 scope.go:122] "RemoveContainer" containerID="832c6a23342dd2effa291183aad3dc0394ced2ab8f2da962c33029590e9eb2b1"
Nov 03 00:00:18.667140 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:18.667113    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft_odh-ai-gateway-infra(e595d849-e5c1-49db-90f3-dc35a24158bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft" podUID="e595d849-e5c1-49db-90f3-dc35a24158bd"
Nov 03 00:00:18.675948 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:18.675924    4294 scope.go:122] "RemoveContainer" containerID="6df65e14caee1c3d84f109ba44d39a1b6356162f234fa650216d485636709921"
Nov 03 00:00:18.693825 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:18.693805    4294 scope.go:122] "RemoveContainer" containerID="efe29c878dfa5dc0b2672e9f48bdb64c346744655e38097786567b0d24c11412"
Nov 03 00:00:18.819089 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:18.819057    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8"]
Nov 03 00:00:18.828510 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:18.828484    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-59f97a-29894400-7qfh8"]
Nov 03 00:00:18.871525 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:18.871488    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv"]
Nov 03 00:00:19.551292 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:19.551240    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:19.552768 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:19.552733    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:19.553953 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:19.553884    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:00:19.618091 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:19.618063    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cvprl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:00:19.618334 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:19.618114    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl" podUID="1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:00:19.660759 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:19.660725    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:19.661514 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:19.661382    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:19.662442 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:19.662419    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:00:19.677041 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:19.677013    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht" event={"ID":"d9c95f99-51cf-4420-bb11-d6022fd906f4","Type":"ContainerStarted","Data":"5a2f676c288a5adddcc943b140c602d8336d11c40981055864658d68aedd0fff"}
Nov 03 00:00:19.679039 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:19.679016    4294 generic.go:356] "Generic (PLEG): container finished" podID="8b200fdb-64bb-40f0-9663-1d7f1e7e3723" containerID="9e868b77e8f538060bed818bfafc6bcf62e6eb47aa41881293134985c5e4dab8" exitCode=60
Nov 03 00:00:19.679154 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:19.679081    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx" event={"ID":"8b200fdb-64bb-40f0-9663-1d7f1e7e3723","Type":"ContainerDied","Data":"9e868b77e8f538060bed818bfafc6bcf62e6eb47aa41881293134985c5e4dab8"}
Nov 03 00:00:19.679154 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:19.679136    4294 scope.go:122] "RemoveContainer" containerID="a451e7ffab922be199a83ed64aa22147d6df312a53f23dd08b22e384958e080e"
Nov 03 00:00:19.679398 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:19.679369    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:19.679771 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:19.679579    4294 scope.go:122] "RemoveContainer" containerID="9e868b77e8f538060bed818bfafc6bcf62e6eb47aa41881293134985c5e4dab8"
Nov 03 00:00:19.679927 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:19.679906    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx_odh-ai-gateway-infra(8b200fdb-64bb-40f0-9663-1d7f1e7e3723)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx" podUID="8b200fdb-64bb-40f0-9663-1d7f1e7e3723"
Nov 03 00:00:19.854588 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:19.854561    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv"
Nov 03 00:00:19.915306 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:19.915264    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b27971-4ed6-44d2-972a-b5596ab2fffb-kube-api-access-jbd6z\" (UniqueName: \"kubernetes.io/projected/50b27971-4ed6-44d2-972a-b5596ab2fffb-kube-api-access-jbd6z\") pod \"50b27971-4ed6-44d2-972a-b5596ab2fffb\" (UID: \"50b27971-4ed6-44d2-972a-b5596ab2fffb\") "
Nov 03 00:00:19.915508 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:19.915461    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b27971-4ed6-44d2-972a-b5596ab2fffb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b27971-4ed6-44d2-972a-b5596ab2fffb-maas-api-service-ca\") pod \"50b27971-4ed6-44d2-972a-b5596ab2fffb\" (UID: \"50b27971-4ed6-44d2-972a-b5596ab2fffb\") "
Nov 03 00:00:19.916203 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:19.916043    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b27971-4ed6-44d2-972a-b5596ab2fffb-maas-api-service-ca" pod "50b27971-4ed6-44d2-972a-b5596ab2fffb" (UID: "50b27971-4ed6-44d2-972a-b5596ab2fffb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:00:19.916203 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:19.916175    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b27971-4ed6-44d2-972a-b5596ab2fffb-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:00:19.917841 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:19.917817    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b27971-4ed6-44d2-972a-b5596ab2fffb-kube-api-access-jbd6z" pod "50b27971-4ed6-44d2-972a-b5596ab2fffb" (UID: "50b27971-4ed6-44d2-972a-b5596ab2fffb"). InnerVolumeSpecName "kube-api-access-jbd6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:00:20.017053 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.017023    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbd6z\" (UniqueName: \"kubernetes.io/projected/50b27971-4ed6-44d2-972a-b5596ab2fffb-kube-api-access-jbd6z\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:00:20.070539 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.070463    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft"]
Nov 03 00:00:20.208403 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.208377    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft"
Nov 03 00:00:20.321533 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.321435    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e595d849-e5c1-49db-90f3-dc35a24158bd-kube-api-access-s6xbb\" (UniqueName: \"kubernetes.io/projected/e595d849-e5c1-49db-90f3-dc35a24158bd-kube-api-access-s6xbb\") pod \"e595d849-e5c1-49db-90f3-dc35a24158bd\" (UID: \"e595d849-e5c1-49db-90f3-dc35a24158bd\") "
Nov 03 00:00:20.321711 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.321568    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e595d849-e5c1-49db-90f3-dc35a24158bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e595d849-e5c1-49db-90f3-dc35a24158bd-maas-api-service-ca\") pod \"e595d849-e5c1-49db-90f3-dc35a24158bd\" (UID: \"e595d849-e5c1-49db-90f3-dc35a24158bd\") "
Nov 03 00:00:20.322589 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.322558    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e595d849-e5c1-49db-90f3-dc35a24158bd-maas-api-service-ca" pod "e595d849-e5c1-49db-90f3-dc35a24158bd" (UID: "e595d849-e5c1-49db-90f3-dc35a24158bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:00:20.325246 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.325218    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e595d849-e5c1-49db-90f3-dc35a24158bd-kube-api-access-s6xbb" pod "e595d849-e5c1-49db-90f3-dc35a24158bd" (UID: "e595d849-e5c1-49db-90f3-dc35a24158bd"). InnerVolumeSpecName "kube-api-access-s6xbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:00:20.423075 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.423045    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6xbb\" (UniqueName: \"kubernetes.io/projected/e595d849-e5c1-49db-90f3-dc35a24158bd-kube-api-access-s6xbb\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:00:20.423075 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.423070    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e595d849-e5c1-49db-90f3-dc35a24158bd-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:00:20.549303 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.549274    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ecf1fa4-ede5-420c-ace8-ef7bfb01423b" path="/var/lib/kubelet/pods/4ecf1fa4-ede5-420c-ace8-ef7bfb01423b/volumes"
Nov 03 00:00:20.690353 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.690276    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv"
Nov 03 00:00:20.690832 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.690275    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv" event={"ID":"50b27971-4ed6-44d2-972a-b5596ab2fffb","Type":"ContainerDied","Data":"42c94d222eec1f61d8e626775d7a83a552668c872ca02d283344b6613b7c95ed"}
Nov 03 00:00:20.690832 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.690426    4294 scope.go:122] "RemoveContainer" containerID="5d695f78e4457913902b73cb56e74e966fbf0475c68d8a6c2108659fb7ec9ca8"
Nov 03 00:00:20.692344 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.692307    4294 generic.go:356] "Generic (PLEG): container finished" podID="d9c95f99-51cf-4420-bb11-d6022fd906f4" containerID="5a2f676c288a5adddcc943b140c602d8336d11c40981055864658d68aedd0fff" exitCode=60
Nov 03 00:00:20.692502 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.692333    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht" event={"ID":"d9c95f99-51cf-4420-bb11-d6022fd906f4","Type":"ContainerDied","Data":"5a2f676c288a5adddcc943b140c602d8336d11c40981055864658d68aedd0fff"}
Nov 03 00:00:20.692771 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.692742    4294 scope.go:122] "RemoveContainer" containerID="5a2f676c288a5adddcc943b140c602d8336d11c40981055864658d68aedd0fff"
Nov 03 00:00:20.693101 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:20.693078    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht_odh-ai-gateway-infra(d9c95f99-51cf-4420-bb11-d6022fd906f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht" podUID="d9c95f99-51cf-4420-bb11-d6022fd906f4"
Nov 03 00:00:20.695655 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.695637    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft"
Nov 03 00:00:20.695747 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.695662    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft" event={"ID":"e595d849-e5c1-49db-90f3-dc35a24158bd","Type":"ContainerDied","Data":"aacde8366f5b383d175326c9f7d19ee58631e11f1e30a1fba0fb2b53b1538a4d"}
Nov 03 00:00:20.701211 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.701188    4294 scope.go:122] "RemoveContainer" containerID="033e342982292681f9ee237654d54d40a22043b24b75a53bb76a517f8df743ba"
Nov 03 00:00:20.713842 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.713824    4294 scope.go:122] "RemoveContainer" containerID="832c6a23342dd2effa291183aad3dc0394ced2ab8f2da962c33029590e9eb2b1"
Nov 03 00:00:20.958080 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.958002    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv"]
Nov 03 00:00:20.961348 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:20.961322    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894400-58nzv"]
Nov 03 00:00:21.010156 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.010123    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht"]
Nov 03 00:00:21.064021 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.063971    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft"]
Nov 03 00:00:21.067305 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.067279    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fbec82-29894400-5d9ft"]
Nov 03 00:00:21.113519 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.113492    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx"]
Nov 03 00:00:21.290926 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.290895    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx"
Nov 03 00:00:21.436664 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.436633    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b200fdb-64bb-40f0-9663-1d7f1e7e3723-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b200fdb-64bb-40f0-9663-1d7f1e7e3723-maas-api-service-ca\") pod \"8b200fdb-64bb-40f0-9663-1d7f1e7e3723\" (UID: \"8b200fdb-64bb-40f0-9663-1d7f1e7e3723\") "
Nov 03 00:00:21.436869 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.436715    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b200fdb-64bb-40f0-9663-1d7f1e7e3723-kube-api-access-5dtgr\" (UniqueName: \"kubernetes.io/projected/8b200fdb-64bb-40f0-9663-1d7f1e7e3723-kube-api-access-5dtgr\") pod \"8b200fdb-64bb-40f0-9663-1d7f1e7e3723\" (UID: \"8b200fdb-64bb-40f0-9663-1d7f1e7e3723\") "
Nov 03 00:00:21.437176 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.437142    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b200fdb-64bb-40f0-9663-1d7f1e7e3723-maas-api-service-ca" pod "8b200fdb-64bb-40f0-9663-1d7f1e7e3723" (UID: "8b200fdb-64bb-40f0-9663-1d7f1e7e3723"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:00:21.437367 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.437348    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b200fdb-64bb-40f0-9663-1d7f1e7e3723-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:00:21.439077 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.439053    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b200fdb-64bb-40f0-9663-1d7f1e7e3723-kube-api-access-5dtgr" pod "8b200fdb-64bb-40f0-9663-1d7f1e7e3723" (UID: "8b200fdb-64bb-40f0-9663-1d7f1e7e3723"). InnerVolumeSpecName "kube-api-access-5dtgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:00:21.538779 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.538700    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dtgr\" (UniqueName: \"kubernetes.io/projected/8b200fdb-64bb-40f0-9663-1d7f1e7e3723-kube-api-access-5dtgr\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:00:21.706388 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.706355    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:21.706388 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.706374    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx" event={"ID":"8b200fdb-64bb-40f0-9663-1d7f1e7e3723","Type":"ContainerDied","Data":"696dacf0e6f0013f0a63b2b12d2ee33815243837589f83c98d8cd9bce103b82c"}
Nov 03 00:00:21.706805 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.706416    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx"
Nov 03 00:00:21.706805 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.706464    4294 scope.go:122] "RemoveContainer" containerID="9e868b77e8f538060bed818bfafc6bcf62e6eb47aa41881293134985c5e4dab8"
Nov 03 00:00:21.865534 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.865494    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht"
Nov 03 00:00:21.923937 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.923904    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx"]
Nov 03 00:00:21.927094 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.927073    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ad18ae-29894400-z8mwx"]
Nov 03 00:00:21.944396 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.944362    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c95f99-51cf-4420-bb11-d6022fd906f4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c95f99-51cf-4420-bb11-d6022fd906f4-maas-api-service-ca\") pod \"d9c95f99-51cf-4420-bb11-d6022fd906f4\" (UID: \"d9c95f99-51cf-4420-bb11-d6022fd906f4\") "
Nov 03 00:00:21.944551 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.944453    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c95f99-51cf-4420-bb11-d6022fd906f4-kube-api-access-wzrlw\" (UniqueName: \"kubernetes.io/projected/d9c95f99-51cf-4420-bb11-d6022fd906f4-kube-api-access-wzrlw\") pod \"d9c95f99-51cf-4420-bb11-d6022fd906f4\" (UID: \"d9c95f99-51cf-4420-bb11-d6022fd906f4\") "
Nov 03 00:00:21.944818 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.944796    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c95f99-51cf-4420-bb11-d6022fd906f4-maas-api-service-ca" pod "d9c95f99-51cf-4420-bb11-d6022fd906f4" (UID: "d9c95f99-51cf-4420-bb11-d6022fd906f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:00:21.945016 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.944967    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c95f99-51cf-4420-bb11-d6022fd906f4-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:00:21.946609 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:21.946588    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c95f99-51cf-4420-bb11-d6022fd906f4-kube-api-access-wzrlw" pod "d9c95f99-51cf-4420-bb11-d6022fd906f4" (UID: "d9c95f99-51cf-4420-bb11-d6022fd906f4"). InnerVolumeSpecName "kube-api-access-wzrlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:00:22.046529 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:22.046491    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzrlw\" (UniqueName: \"kubernetes.io/projected/d9c95f99-51cf-4420-bb11-d6022fd906f4-kube-api-access-wzrlw\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:00:22.548764 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:22.548728    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b27971-4ed6-44d2-972a-b5596ab2fffb" path="/var/lib/kubelet/pods/50b27971-4ed6-44d2-972a-b5596ab2fffb/volumes"
Nov 03 00:00:22.549188 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:22.549174    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b200fdb-64bb-40f0-9663-1d7f1e7e3723" path="/var/lib/kubelet/pods/8b200fdb-64bb-40f0-9663-1d7f1e7e3723/volumes"
Nov 03 00:00:22.549553 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:22.549541    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e595d849-e5c1-49db-90f3-dc35a24158bd" path="/var/lib/kubelet/pods/e595d849-e5c1-49db-90f3-dc35a24158bd/volumes"
Nov 03 00:00:22.715517 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:22.715488    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht"
Nov 03 00:00:22.715949 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:22.715536    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht" event={"ID":"d9c95f99-51cf-4420-bb11-d6022fd906f4","Type":"ContainerDied","Data":"cfc77f94096ee5637df88f045ff2174ab317f35693b5013500df0a0feab85635"}
Nov 03 00:00:22.715949 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:22.715600    4294 scope.go:122] "RemoveContainer" containerID="5a2f676c288a5adddcc943b140c602d8336d11c40981055864658d68aedd0fff"
Nov 03 00:00:22.873456 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:22.873367    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht"]
Nov 03 00:00:22.920121 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:22.920088    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-fe63ac-29894400-lckht"]
Nov 03 00:00:23.241889 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:23.241852    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:00:23.638348 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:23.638264    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:24.377599 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:24.377568    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:24.549339 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:24.549307    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9c95f99-51cf-4420-bb11-d6022fd906f4" path="/var/lib/kubelet/pods/d9c95f99-51cf-4420-bb11-d6022fd906f4/volumes"
Nov 03 00:00:24.862301 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:24.862262    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:25.832218 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:25.832175    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:26.007605 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:26.007568    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:28.978361 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:28.978317    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:30.087335 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:30.087286    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:30.278032 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:30.277958    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:30.446818 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:30.446713    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:33.242539 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.242494    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:00:33.277573 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.277484    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:33.447642 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.447605    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:33.788843 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.788807    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh"]
Nov 03 00:00:33.789797 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.789776    4294 cpu_manager.go:414] "containerMap: removing container" podUID="8b200fdb-64bb-40f0-9663-1d7f1e7e3723" containerName="cleanup"
Nov 03 00:00:33.789797 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.789796    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b200fdb-64bb-40f0-9663-1d7f1e7e3723" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.789810    4294 cpu_manager.go:414] "containerMap: removing container" podUID="50b27971-4ed6-44d2-972a-b5596ab2fffb" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.789818    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b27971-4ed6-44d2-972a-b5596ab2fffb" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.789838    4294 cpu_manager.go:414] "containerMap: removing container" podUID="d9c95f99-51cf-4420-bb11-d6022fd906f4" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.789846    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c95f99-51cf-4420-bb11-d6022fd906f4" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.789870    4294 cpu_manager.go:414] "containerMap: removing container" podUID="8b200fdb-64bb-40f0-9663-1d7f1e7e3723" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.789877    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b200fdb-64bb-40f0-9663-1d7f1e7e3723" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.789889    4294 cpu_manager.go:414] "containerMap: removing container" podUID="50b27971-4ed6-44d2-972a-b5596ab2fffb" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.789896    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b27971-4ed6-44d2-972a-b5596ab2fffb" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.789912    4294 cpu_manager.go:414] "containerMap: removing container" podUID="50b27971-4ed6-44d2-972a-b5596ab2fffb" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.789919    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b27971-4ed6-44d2-972a-b5596ab2fffb" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.789930    4294 cpu_manager.go:414] "containerMap: removing container" podUID="4ecf1fa4-ede5-420c-ace8-ef7bfb01423b" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.789936    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ecf1fa4-ede5-420c-ace8-ef7bfb01423b" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.789945    4294 cpu_manager.go:414] "containerMap: removing container" podUID="e595d849-e5c1-49db-90f3-dc35a24158bd" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.789951    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e595d849-e5c1-49db-90f3-dc35a24158bd" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.789965    4294 cpu_manager.go:414] "containerMap: removing container" podUID="e595d849-e5c1-49db-90f3-dc35a24158bd" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.789972    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e595d849-e5c1-49db-90f3-dc35a24158bd" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.790016    4294 cpu_manager.go:414] "containerMap: removing container" podUID="d9c95f99-51cf-4420-bb11-d6022fd906f4" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.790022    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c95f99-51cf-4420-bb11-d6022fd906f4" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.790031    4294 cpu_manager.go:414] "containerMap: removing container" podUID="4ecf1fa4-ede5-420c-ace8-ef7bfb01423b" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.790038    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ecf1fa4-ede5-420c-ace8-ef7bfb01423b" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.790046    4294 cpu_manager.go:414] "containerMap: removing container" podUID="8b200fdb-64bb-40f0-9663-1d7f1e7e3723" containerName="cleanup"
Nov 03 00:00:33.790064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.790053    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b200fdb-64bb-40f0-9663-1d7f1e7e3723" containerName="cleanup"
Nov 03 00:00:33.790834 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.790197    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ecf1fa4-ede5-420c-ace8-ef7bfb01423b" containerName="cleanup"
Nov 03 00:00:33.790834 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.790208    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b200fdb-64bb-40f0-9663-1d7f1e7e3723" containerName="cleanup"
Nov 03 00:00:33.790834 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.790223    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b27971-4ed6-44d2-972a-b5596ab2fffb" containerName="cleanup"
Nov 03 00:00:33.790834 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.790235    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c95f99-51cf-4420-bb11-d6022fd906f4" containerName="cleanup"
Nov 03 00:00:33.790834 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.790246    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b200fdb-64bb-40f0-9663-1d7f1e7e3723" containerName="cleanup"
Nov 03 00:00:33.790834 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.790255    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b27971-4ed6-44d2-972a-b5596ab2fffb" containerName="cleanup"
Nov 03 00:00:33.790834 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.790264    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b27971-4ed6-44d2-972a-b5596ab2fffb" containerName="cleanup"
Nov 03 00:00:33.790834 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.790274    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e595d849-e5c1-49db-90f3-dc35a24158bd" containerName="cleanup"
Nov 03 00:00:33.790834 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.790285    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e595d849-e5c1-49db-90f3-dc35a24158bd" containerName="cleanup"
Nov 03 00:00:33.790834 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.790294    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e595d849-e5c1-49db-90f3-dc35a24158bd" containerName="cleanup"
Nov 03 00:00:33.790834 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.790303    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c95f99-51cf-4420-bb11-d6022fd906f4" containerName="cleanup"
Nov 03 00:00:33.790834 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.790313    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ecf1fa4-ede5-420c-ace8-ef7bfb01423b" containerName="cleanup"
Nov 03 00:00:33.796451 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.796427    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh"
Nov 03 00:00:33.843966 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.843924    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh"]
Nov 03 00:00:33.886035 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.886005    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-666a38\""
Nov 03 00:00:33.886214 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.886047    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-666a38-dockercfg-62m5c\""
Nov 03 00:00:33.916705 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.916669    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdcm4\" (UniqueName: \"kubernetes.io/projected/c2d57a6a-c210-4895-8124-53a2514d1d3c-kube-api-access-rdcm4\") pod \"payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh\" (UID: \"c2d57a6a-c210-4895-8124-53a2514d1d3c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh"
Nov 03 00:00:33.916887 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:33.916833    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d57a6a-c210-4895-8124-53a2514d1d3c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh\" (UID: \"c2d57a6a-c210-4895-8124-53a2514d1d3c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh"
Nov 03 00:00:34.006685 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:34.006652    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:34.018358 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:34.018327    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d57a6a-c210-4895-8124-53a2514d1d3c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh\" (UID: \"c2d57a6a-c210-4895-8124-53a2514d1d3c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh"
Nov 03 00:00:34.018563 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:34.018547    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdcm4\" (UniqueName: \"kubernetes.io/projected/c2d57a6a-c210-4895-8124-53a2514d1d3c-kube-api-access-rdcm4\") pod \"payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh\" (UID: \"c2d57a6a-c210-4895-8124-53a2514d1d3c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh"
Nov 03 00:00:34.019067 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:34.019046    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d57a6a-c210-4895-8124-53a2514d1d3c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh\" (UID: \"c2d57a6a-c210-4895-8124-53a2514d1d3c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh"
Nov 03 00:00:34.073330 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:34.073259    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdcm4\" (UniqueName: \"kubernetes.io/projected/c2d57a6a-c210-4895-8124-53a2514d1d3c-kube-api-access-rdcm4\") pod \"payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh\" (UID: \"c2d57a6a-c210-4895-8124-53a2514d1d3c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh"
Nov 03 00:00:34.111779 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:34.111743    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh"
Nov 03 00:00:34.478368 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:00:34.478331    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d57a6a_c210_4895_8124_53a2514d1d3c.slice/crio-75eb5ea37aad40515677a64ac92d06a3cf9b74e5cd23a049d4bcea0b752ea740 WatchSource:0}: Error finding container 75eb5ea37aad40515677a64ac92d06a3cf9b74e5cd23a049d4bcea0b752ea740: Status 404 returned error can't find the container with id 75eb5ea37aad40515677a64ac92d06a3cf9b74e5cd23a049d4bcea0b752ea740
Nov 03 00:00:34.480293 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:34.480266    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh"]
Nov 03 00:00:34.803409 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:34.803376    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh" event={"ID":"c2d57a6a-c210-4895-8124-53a2514d1d3c","Type":"ContainerStarted","Data":"44840256b8d07665a8d82be1157ad8f89b8385bf13e21a70a35d101c713332a8"}
Nov 03 00:00:34.803620 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:34.803435    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh" event={"ID":"c2d57a6a-c210-4895-8124-53a2514d1d3c","Type":"ContainerStarted","Data":"75eb5ea37aad40515677a64ac92d06a3cf9b74e5cd23a049d4bcea0b752ea740"}
Nov 03 00:00:34.803620 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:34.803461    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh"
Nov 03 00:00:34.914357 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:34.914301    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh" podStartSLOduration=1.914286972 podStartE2EDuration="1.914286972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:34.911812458 +0000 UTC m=+4074.480167245" watchObservedRunningTime="2026-11-03 00:00:34.914286972 +0000 UTC m=+4074.482641759"
Nov 03 00:00:38.005968 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:38.005935    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:38.423747 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:38.423643    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:38.591632 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:38.591600    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:38.747490 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:38.747457    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:38.981908 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:38.981876    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:39.160642 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:39.160560    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:39.523545 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:39.523511    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:41.756156 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:41.756115    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:00:43.109659 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:43.109626    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:43.242539 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:43.242492    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:00:43.658461 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:43.658429    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:43.967085 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:43.967009    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:44.177416 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:44.177384    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:44.403684 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:44.403650    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:44.627724 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:44.627685    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:45.060742 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:45.060699    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:45.225264 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:45.225229    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:45.813211 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:45.813180    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh"
Nov 03 00:00:48.587653 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:48.587618    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:48.924116 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:48.924033    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:49.088540 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.088511    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:49.295228 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.295188    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq"]
Nov 03 00:00:49.295929 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.295906    4294 cpu_manager.go:414] "containerMap: removing container" podUID="d9c95f99-51cf-4420-bb11-d6022fd906f4" containerName="cleanup"
Nov 03 00:00:49.295929 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.295928    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c95f99-51cf-4420-bb11-d6022fd906f4" containerName="cleanup"
Nov 03 00:00:49.296166 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.296009    4294 cpu_manager.go:414] "containerMap: removing container" podUID="4ecf1fa4-ede5-420c-ace8-ef7bfb01423b" containerName="cleanup"
Nov 03 00:00:49.296166 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.296019    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ecf1fa4-ede5-420c-ace8-ef7bfb01423b" containerName="cleanup"
Nov 03 00:00:49.296166 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.296033    4294 cpu_manager.go:414] "containerMap: removing container" podUID="e595d849-e5c1-49db-90f3-dc35a24158bd" containerName="cleanup"
Nov 03 00:00:49.296166 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.296041    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e595d849-e5c1-49db-90f3-dc35a24158bd" containerName="cleanup"
Nov 03 00:00:49.296400 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.296207    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ecf1fa4-ede5-420c-ace8-ef7bfb01423b" containerName="cleanup"
Nov 03 00:00:49.296400 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.296221    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b200fdb-64bb-40f0-9663-1d7f1e7e3723" containerName="cleanup"
Nov 03 00:00:49.296400 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.296246    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c95f99-51cf-4420-bb11-d6022fd906f4" containerName="cleanup"
Nov 03 00:00:49.302213 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.302193    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq"
Nov 03 00:00:49.317313 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.316129    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b964a56-7099-408e-97dd-71a707fdacbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq\" (UID: \"4b964a56-7099-408e-97dd-71a707fdacbd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq"
Nov 03 00:00:49.317313 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.316222    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmp7x\" (UniqueName: \"kubernetes.io/projected/4b964a56-7099-408e-97dd-71a707fdacbd-kube-api-access-cmp7x\") pod \"payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq\" (UID: \"4b964a56-7099-408e-97dd-71a707fdacbd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq"
Nov 03 00:00:49.350801 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.350774    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq"]
Nov 03 00:00:49.394682 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.394639    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c9d4fc\""
Nov 03 00:00:49.394682 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.394644    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c9d4fc-dockercfg-ncqdc\""
Nov 03 00:00:49.417187 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.417153    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b964a56-7099-408e-97dd-71a707fdacbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq\" (UID: \"4b964a56-7099-408e-97dd-71a707fdacbd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq"
Nov 03 00:00:49.417380 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.417216    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmp7x\" (UniqueName: \"kubernetes.io/projected/4b964a56-7099-408e-97dd-71a707fdacbd-kube-api-access-cmp7x\") pod \"payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq\" (UID: \"4b964a56-7099-408e-97dd-71a707fdacbd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq"
Nov 03 00:00:49.417832 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.417813    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b964a56-7099-408e-97dd-71a707fdacbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq\" (UID: \"4b964a56-7099-408e-97dd-71a707fdacbd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq"
Nov 03 00:00:49.471439 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.471405    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmp7x\" (UniqueName: \"kubernetes.io/projected/4b964a56-7099-408e-97dd-71a707fdacbd-kube-api-access-cmp7x\") pod \"payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq\" (UID: \"4b964a56-7099-408e-97dd-71a707fdacbd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq"
Nov 03 00:00:49.626718 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:49.626642    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq"
Nov 03 00:00:50.000165 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:50.000134    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq"]
Nov 03 00:00:50.001038 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:00:50.001013    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b964a56_7099_408e_97dd_71a707fdacbd.slice/crio-2d88500f1e4b7a317b2a36553c07e3b38139ad2d8937475cb44c75c4656bd654 WatchSource:0}: Error finding container 2d88500f1e4b7a317b2a36553c07e3b38139ad2d8937475cb44c75c4656bd654: Status 404 returned error can't find the container with id 2d88500f1e4b7a317b2a36553c07e3b38139ad2d8937475cb44c75c4656bd654
Nov 03 00:00:50.917119 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:50.917088    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq" event={"ID":"4b964a56-7099-408e-97dd-71a707fdacbd","Type":"ContainerStarted","Data":"acdf3817fbf863fe43db5b952d067797348e3b35eb5fa22100c622f33010e7d9"}
Nov 03 00:00:50.917623 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:50.917137    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq" event={"ID":"4b964a56-7099-408e-97dd-71a707fdacbd","Type":"ContainerStarted","Data":"2d88500f1e4b7a317b2a36553c07e3b38139ad2d8937475cb44c75c4656bd654"}
Nov 03 00:00:50.917623 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:50.917200    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq"
Nov 03 00:00:51.027437 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:51.027383    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq" podStartSLOduration=2.027365928 podStartE2EDuration="2.027365928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:00:51.026838673 +0000 UTC m=+4090.595193460" watchObservedRunningTime="2026-11-03 00:00:51.027365928 +0000 UTC m=+4090.595720716"
Nov 03 00:00:53.242536 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:53.242498    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:00:53.425469 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:53.425433    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:00:53.466366 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:53.466334    4294 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a610cb5fdd81: secret "agc-cred-a610cb5fdd81" not found
Nov 03 00:00:53.466366 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:53.466364    4294 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb3055757321: secret "agc-cred-eb3055757321" not found
Nov 03 00:00:53.466366 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:53.466374    4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-64887bb858-xh446: [secret "agc-cred-a610cb5fdd81" not found, secret "agc-cred-eb3055757321" not found]
Nov 03 00:00:53.466646 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:00:53.466441    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials podName:6ae49deb-156f-4f33-a81d-06bc93b7dcb8 nodeName:}" failed. No retries permitted until 2026-11-03 00:02:55.466423713 +0000 UTC m=+4215.034778478 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials") pod "payload-processing-external-model-64887bb858-xh446" (UID: "6ae49deb-156f-4f33-a81d-06bc93b7dcb8") : [secret "agc-cred-a610cb5fdd81" not found, secret "agc-cred-eb3055757321" not found]
Nov 03 00:00:57.957069 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:00:57.957033    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:01:01.928164 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:01.928134    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq"
Nov 03 00:01:02.447755 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:02.447723    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:01:02.542390 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:02.542345    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:01:06.760797 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:06.760758    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"]
Nov 03 00:01:06.765583 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:06.765561    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"
Nov 03 00:01:06.818222 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:06.818188    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"]
Nov 03 00:01:06.861290 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:06.861261    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1f29e6-serving-cert\""
Nov 03 00:01:06.958936 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:06.958896    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ffb04c4-4f77-4da0-b266-b72574803dd0-metrics-certs\") pod \"maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv\" (UID: \"7ffb04c4-4f77-4da0-b266-b72574803dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"
Nov 03 00:01:06.959144 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:06.958962    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ffb04c4-4f77-4da0-b266-b72574803dd0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv\" (UID: \"7ffb04c4-4f77-4da0-b266-b72574803dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"
Nov 03 00:01:06.959144 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:06.959101    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ffb04c4-4f77-4da0-b266-b72574803dd0-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv\" (UID: \"7ffb04c4-4f77-4da0-b266-b72574803dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"
Nov 03 00:01:06.959144 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:06.959138    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzgtz\" (UniqueName: \"kubernetes.io/projected/7ffb04c4-4f77-4da0-b266-b72574803dd0-kube-api-access-rzgtz\") pod \"maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv\" (UID: \"7ffb04c4-4f77-4da0-b266-b72574803dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"
Nov 03 00:01:06.986673 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:06.986639    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:01:07.060613 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:07.060505    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ffb04c4-4f77-4da0-b266-b72574803dd0-metrics-certs\") pod \"maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv\" (UID: \"7ffb04c4-4f77-4da0-b266-b72574803dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"
Nov 03 00:01:07.060613 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:07.060586    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ffb04c4-4f77-4da0-b266-b72574803dd0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv\" (UID: \"7ffb04c4-4f77-4da0-b266-b72574803dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"
Nov 03 00:01:07.060884 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:07.060635    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ffb04c4-4f77-4da0-b266-b72574803dd0-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv\" (UID: \"7ffb04c4-4f77-4da0-b266-b72574803dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"
Nov 03 00:01:07.060884 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:07.060662    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzgtz\" (UniqueName: \"kubernetes.io/projected/7ffb04c4-4f77-4da0-b266-b72574803dd0-kube-api-access-rzgtz\") pod \"maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv\" (UID: \"7ffb04c4-4f77-4da0-b266-b72574803dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"
Nov 03 00:01:07.061332 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:07.061308    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ffb04c4-4f77-4da0-b266-b72574803dd0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv\" (UID: \"7ffb04c4-4f77-4da0-b266-b72574803dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"
Nov 03 00:01:07.063143 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:07.063123    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ffb04c4-4f77-4da0-b266-b72574803dd0-maas-api-tls\") pod \"maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv\" (UID: \"7ffb04c4-4f77-4da0-b266-b72574803dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"
Nov 03 00:01:07.063199 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:07.063164    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ffb04c4-4f77-4da0-b266-b72574803dd0-metrics-certs\") pod \"maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv\" (UID: \"7ffb04c4-4f77-4da0-b266-b72574803dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"
Nov 03 00:01:07.114960 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:07.114931    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzgtz\" (UniqueName: \"kubernetes.io/projected/7ffb04c4-4f77-4da0-b266-b72574803dd0-kube-api-access-rzgtz\") pod \"maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv\" (UID: \"7ffb04c4-4f77-4da0-b266-b72574803dd0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"
Nov 03 00:01:07.154907 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:07.154861    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:01:07.380407 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:07.380302    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"
Nov 03 00:01:07.755355 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:01:07.755309    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ffb04c4_4f77_4da0_b266_b72574803dd0.slice/crio-05adc7c163dab074086beff5d15df0e4c0850f1a577884e3244a2e2e68112aa2 WatchSource:0}: Error finding container 05adc7c163dab074086beff5d15df0e4c0850f1a577884e3244a2e2e68112aa2: Status 404 returned error can't find the container with id 05adc7c163dab074086beff5d15df0e4c0850f1a577884e3244a2e2e68112aa2
Nov 03 00:01:07.757126 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:07.757101    4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:01:07.758738 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:07.758707    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"]
Nov 03 00:01:08.031379 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:08.031277    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv" event={"ID":"7ffb04c4-4f77-4da0-b266-b72574803dd0","Type":"ContainerStarted","Data":"05adc7c163dab074086beff5d15df0e4c0850f1a577884e3244a2e2e68112aa2"}
Nov 03 00:01:09.040040 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:09.039974    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv" event={"ID":"7ffb04c4-4f77-4da0-b266-b72574803dd0","Type":"ContainerStarted","Data":"1b428996af04f205336b268bc39c445a1e93a6e48a6616b0a7f02ccf06be5bbd"}
Nov 03 00:01:09.040475 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:09.040062    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"
Nov 03 00:01:09.151992 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:09.151923    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv" podStartSLOduration=2.275086859 podStartE2EDuration="3.151908566s" totalImagesPullingTime="876.821707ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:01:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:01:09.149015447 +0000 UTC m=+4108.717370236" watchObservedRunningTime="2026-11-03 00:01:09.151908566 +0000 UTC m=+4108.720263353"
Nov 03 00:01:11.754649 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:01:11.754608    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:01:11.867917 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:11.867882    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:01:12.543181 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:12.543137    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:01:15.054803 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:15.054769    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"
Nov 03 00:01:16.362549 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:16.362515    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:01:21.410572 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:21.410532    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:01:22.542479 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:22.542435    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:01:23.234513 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:23.234478    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:01:23.391375 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:23.391326    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:01:26.036435 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:26.036401    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:01:30.583600 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:30.583557    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:01:32.542449 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:32.542409    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:01:35.148065 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:35.148033    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:01:39.870113 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:39.870076    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:01:41.751478 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:01:41.751429    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:01:42.542452 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:42.542413    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:01:44.432744 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:44.432706    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:01:44.950448 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:44.950410    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:01:49.044824 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:49.044776    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:01:52.543201 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:52.543158    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:01:53.531030 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:53.530967    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:01:53.693314 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:53.693281    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:01:53.859244 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:53.859152    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:01:58.285873 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:01:58.285837    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:02.542165 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:02.542121    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:02:02.838630 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:02.838538    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:07.369753 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:07.369719    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:11.567469 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:11.567431    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"]
Nov 03 00:02:11.572097 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:11.572068    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"
Nov 03 00:02:11.625634 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:11.625603    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"]
Nov 03 00:02:11.669136 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:11.669109    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c9d4fc-serving-cert\""
Nov 03 00:02:11.679792 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:11.679769    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c764fb-09a2-4b5e-8588-b463f9487461-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2\" (UID: \"a6c764fb-09a2-4b5e-8588-b463f9487461\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"
Nov 03 00:02:11.679915 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:11.679817    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s568v\" (UniqueName: \"kubernetes.io/projected/a6c764fb-09a2-4b5e-8588-b463f9487461-kube-api-access-s568v\") pod \"maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2\" (UID: \"a6c764fb-09a2-4b5e-8588-b463f9487461\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"
Nov 03 00:02:11.679964 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:11.679929    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6c764fb-09a2-4b5e-8588-b463f9487461-maas-api-tls\") pod \"maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2\" (UID: \"a6c764fb-09a2-4b5e-8588-b463f9487461\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"
Nov 03 00:02:11.680025 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:11.679991    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c764fb-09a2-4b5e-8588-b463f9487461-metrics-certs\") pod \"maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2\" (UID: \"a6c764fb-09a2-4b5e-8588-b463f9487461\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"
Nov 03 00:02:11.751103 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:02:11.751064    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:02:11.780287 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:11.780255    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:11.780903 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:11.780884    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c764fb-09a2-4b5e-8588-b463f9487461-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2\" (UID: \"a6c764fb-09a2-4b5e-8588-b463f9487461\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"
Nov 03 00:02:11.780956 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:11.780930    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s568v\" (UniqueName: \"kubernetes.io/projected/a6c764fb-09a2-4b5e-8588-b463f9487461-kube-api-access-s568v\") pod \"maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2\" (UID: \"a6c764fb-09a2-4b5e-8588-b463f9487461\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"
Nov 03 00:02:11.781020 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:11.780972    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6c764fb-09a2-4b5e-8588-b463f9487461-maas-api-tls\") pod \"maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2\" (UID: \"a6c764fb-09a2-4b5e-8588-b463f9487461\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"
Nov 03 00:02:11.781066 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:11.781036    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c764fb-09a2-4b5e-8588-b463f9487461-metrics-certs\") pod \"maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2\" (UID: \"a6c764fb-09a2-4b5e-8588-b463f9487461\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"
Nov 03 00:02:11.781169 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:02:11.781154    4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-serving-cert: secret "maas-api-e2e-inf-b-c9d4fc-serving-cert" not found
Nov 03 00:02:11.781234 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:02:11.781224    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6c764fb-09a2-4b5e-8588-b463f9487461-maas-api-tls podName:a6c764fb-09a2-4b5e-8588-b463f9487461 nodeName:}" failed. No retries permitted until 2026-11-03 00:02:12.281208879 +0000 UTC m=+4171.849563655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a6c764fb-09a2-4b5e-8588-b463f9487461-maas-api-tls") pod "maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2" (UID: "a6c764fb-09a2-4b5e-8588-b463f9487461") : secret "maas-api-e2e-inf-b-c9d4fc-serving-cert" not found
Nov 03 00:02:11.781544 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:11.781517    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c764fb-09a2-4b5e-8588-b463f9487461-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2\" (UID: \"a6c764fb-09a2-4b5e-8588-b463f9487461\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"
Nov 03 00:02:11.783454 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:11.783432    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c764fb-09a2-4b5e-8588-b463f9487461-metrics-certs\") pod \"maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2\" (UID: \"a6c764fb-09a2-4b5e-8588-b463f9487461\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"
Nov 03 00:02:11.836104 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:11.836029    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s568v\" (UniqueName: \"kubernetes.io/projected/a6c764fb-09a2-4b5e-8588-b463f9487461-kube-api-access-s568v\") pod \"maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2\" (UID: \"a6c764fb-09a2-4b5e-8588-b463f9487461\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"
Nov 03 00:02:11.940195 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:11.940155    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:12.102840 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:12.102768    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:12.287899 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:12.287866    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6c764fb-09a2-4b5e-8588-b463f9487461-maas-api-tls\") pod \"maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2\" (UID: \"a6c764fb-09a2-4b5e-8588-b463f9487461\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"
Nov 03 00:02:12.290610 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:12.290583    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6c764fb-09a2-4b5e-8588-b463f9487461-maas-api-tls\") pod \"maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2\" (UID: \"a6c764fb-09a2-4b5e-8588-b463f9487461\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"
Nov 03 00:02:12.485897 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:12.485813    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"
Nov 03 00:02:12.542768 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:12.542740    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:02:12.858658 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:02:12.858628    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c764fb_09a2_4b5e_8588_b463f9487461.slice/crio-802572823f6f76256f3911792730b3e71e08a1a2cf7e3266ccd7df4e28700e7b WatchSource:0}: Error finding container 802572823f6f76256f3911792730b3e71e08a1a2cf7e3266ccd7df4e28700e7b: Status 404 returned error can't find the container with id 802572823f6f76256f3911792730b3e71e08a1a2cf7e3266ccd7df4e28700e7b
Nov 03 00:02:12.859027 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:12.858961    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"]
Nov 03 00:02:13.487403 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:13.487368    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2" event={"ID":"a6c764fb-09a2-4b5e-8588-b463f9487461","Type":"ContainerStarted","Data":"802572823f6f76256f3911792730b3e71e08a1a2cf7e3266ccd7df4e28700e7b"}
Nov 03 00:02:14.496035 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:14.496001    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2" event={"ID":"a6c764fb-09a2-4b5e-8588-b463f9487461","Type":"ContainerStarted","Data":"a31ba681616b0337db5344c4c4b6c0766ac5fa6078e1d65646b432e723c86838"}
Nov 03 00:02:14.496427 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:14.496214    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"
Nov 03 00:02:14.619479 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:14.615645    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2" podStartSLOduration=2.777805968 podStartE2EDuration="3.615623927s" totalImagesPullingTime="837.817959ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:02:14.605752887 +0000 UTC m=+4174.174107692" watchObservedRunningTime="2026-11-03 00:02:14.615623927 +0000 UTC m=+4174.183978715"
Nov 03 00:02:16.280013 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:16.279950    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:16.751548 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:16.751515    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:16.957865 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:16.957834    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:17.364022 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:17.363971    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:17.685605 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:17.685526    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:20.508305 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:20.508275    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"
Nov 03 00:02:21.542022 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:21.541963    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:02:21.946676 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:21.946597    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:22.309482 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:22.309450    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:22.499042 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:22.498974    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:22.670131 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:22.670044    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:25.471660 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:25.471586    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:25.632930 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:25.632898    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:26.861550 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:26.861515    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"]
Nov 03 00:02:26.865852 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:26.865836    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:26.899485 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:26.899454    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff7037f-31de-4b57-a263-0229aa21b689-istiod-ca-cert\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:26.899639 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:26.899523    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-istio-data\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:26.899711 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:26.899631    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-credential-socket\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:26.899711 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:26.899673    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-workload-certs\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:26.899825 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:26.899790    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff7037f-31de-4b57-a263-0229aa21b689-istio-token\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:26.899825 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:26.899812    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-istio-envoy\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:26.899918 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:26.899892    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-workload-socket\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:26.899970 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:26.899924    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff7037f-31de-4b57-a263-0229aa21b689-istio-podinfo\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:26.900118 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:26.900101    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pplxx\" (UniqueName: \"kubernetes.io/projected/aff7037f-31de-4b57-a263-0229aa21b689-kube-api-access-pplxx\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:26.916325 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:26.916301    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-666a38-openshift-default-dockercfg-xbjck\""
Nov 03 00:02:26.968021 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:26.967972    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"]
Nov 03 00:02:27.001495 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.001464    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pplxx\" (UniqueName: \"kubernetes.io/projected/aff7037f-31de-4b57-a263-0229aa21b689-kube-api-access-pplxx\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:27.001673 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.001516    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff7037f-31de-4b57-a263-0229aa21b689-istiod-ca-cert\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:27.001673 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.001573    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-istio-data\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:27.001673 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.001601    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-credential-socket\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:27.001673 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.001628    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-workload-certs\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:27.001920 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.001755    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff7037f-31de-4b57-a263-0229aa21b689-istio-token\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:27.001920 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.001786    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-istio-envoy\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:27.001920 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.001845    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-workload-socket\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:27.001920 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.001883    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff7037f-31de-4b57-a263-0229aa21b689-istio-podinfo\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:27.002184 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.002091    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-istio-data\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:27.002184 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.002157    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-credential-socket\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:27.002259 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.002203    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-workload-certs\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:27.002349 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.002327    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-workload-socket\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:27.002390 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.002375    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff7037f-31de-4b57-a263-0229aa21b689-istiod-ca-cert\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:27.004076 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.004056    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-istio-envoy\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:27.004321 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.004303    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff7037f-31de-4b57-a263-0229aa21b689-istio-podinfo\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:27.058585 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.058544    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pplxx\" (UniqueName: \"kubernetes.io/projected/aff7037f-31de-4b57-a263-0229aa21b689-kube-api-access-pplxx\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:27.059061 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.059042    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff7037f-31de-4b57-a263-0229aa21b689-istio-token\") pod \"e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") " pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:27.124181 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.124032    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:27.179919 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.179887    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:27.618794 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.618749    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"]
Nov 03 00:02:27.619203 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.619161    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:02:27.619336 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.619309    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:02:27.619436 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:27.619411    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:02:28.616225 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:28.616189    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt" event={"ID":"aff7037f-31de-4b57-a263-0229aa21b689","Type":"ContainerStarted","Data":"9a49544380cba41eaa1a865959bcf5e224d97928daa9ac92df8d8bd1cf719ec0"}
Nov 03 00:02:28.616654 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:28.616255    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt" event={"ID":"aff7037f-31de-4b57-a263-0229aa21b689","Type":"ContainerStarted","Data":"b87180287c29bfaac5abf43566aa77973038f8f0793d0cab58de6230b3cfa2d3"}
Nov 03 00:02:28.838116 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:28.838063    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt" podStartSLOduration=2.8380470840000003 podStartE2EDuration="2.838047084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:02:28.790202155 +0000 UTC m=+4188.358556943" watchObservedRunningTime="2026-11-03 00:02:28.838047084 +0000 UTC m=+4188.406401871"
Nov 03 00:02:29.180923 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:29.180882    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:29.182734 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:29.182702    4294 patch_prober.go:29] interesting pod/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 00:02:29.182889 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:29.182762    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt" podUID="aff7037f-31de-4b57-a263-0229aa21b689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 00:02:30.180718 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:30.180681    4294 patch_prober.go:29] interesting pod/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 00:02:30.181127 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:30.180748    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt" podUID="aff7037f-31de-4b57-a263-0229aa21b689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 00:02:31.181174 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:31.181132    4294 patch_prober.go:29] interesting pod/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 00:02:31.181611 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:31.181187    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt" podUID="aff7037f-31de-4b57-a263-0229aa21b689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 00:02:31.542383 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:31.542340    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:02:32.180277 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:32.180240    4294 patch_prober.go:29] interesting pod/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 00:02:32.180476 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:32.180300    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt" podUID="aff7037f-31de-4b57-a263-0229aa21b689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 00:02:32.212058 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:32.212011    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:33.181208 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:33.181173    4294 patch_prober.go:29] interesting pod/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 00:02:33.181402 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:33.181228    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt" podUID="aff7037f-31de-4b57-a263-0229aa21b689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 00:02:34.185682 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:34.185650    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:34.186149 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:34.186128    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:34.186876 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:34.186858    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:02:37.152257 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:37.152228    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:41.542412 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:41.542369    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:02:41.718525 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:41.718485    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:41.757000 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:02:41.756265    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:02:45.318287 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:45.318247    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:47.475143 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:47.475093    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:50.267821 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:50.267784    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:51.542130 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:51.542084    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:02:51.611961 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:51.611905    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:55.510283 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:02:55.510250    4294 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a610cb5fdd81: secret "agc-cred-a610cb5fdd81" not found
Nov 03 00:02:55.510283 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:02:55.510281    4294 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb3055757321: secret "agc-cred-eb3055757321" not found
Nov 03 00:02:55.510283 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:02:55.510291    4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-64887bb858-xh446: [secret "agc-cred-a610cb5fdd81" not found, secret "agc-cred-eb3055757321" not found]
Nov 03 00:02:55.510863 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:02:55.510363    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials podName:6ae49deb-156f-4f33-a81d-06bc93b7dcb8 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:57.51034846 +0000 UTC m=+4337.078703225 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials") pod "payload-processing-external-model-64887bb858-xh446" (UID: "6ae49deb-156f-4f33-a81d-06bc93b7dcb8") : [secret "agc-cred-a610cb5fdd81" not found, secret "agc-cred-eb3055757321" not found]
Nov 03 00:02:55.963463 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:55.963348    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:56.554341 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:56.554288    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:57.295549 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:57.295513    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj"]
Nov 03 00:02:57.398441 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:57.398403    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6"]
Nov 03 00:02:57.682231 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:57.682149    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:02:57.850076 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:02:57.850040    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:01.279716 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:01.279597    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:01.542109 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:01.542025    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:03:01.691894 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:01.691863    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:02.295840 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.295796    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj" podUID="d36c485f-2e64-48e6-ae55-df84f85dd47b" containerName="payload-processing" containerID="cri-o://83ff5d2d98bdddd2b77e21c42c75b6861ad91e3c270563ace06302b936b27593" gracePeriod=25
Nov 03 00:03:02.307715 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:03:02.307687    4294 logging.go:55] [core] [Channel #16731 SubChannel #16733]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.185:50052", ServerName: "10.133.0.185:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.185:50052: connect: connection refused"
Nov 03 00:03:02.399084 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.399043    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6" podUID="bb92a5d9-abf6-4d20-9919-5edce78d957f" containerName="payload-pre-processing" containerID="cri-o://05c49aa21512b56e110ff6513234ebfe9026482b3d3c875d5cde1cf9edfda65b" gracePeriod=25
Nov 03 00:03:02.601743 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.601710    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj"
Nov 03 00:03:02.693721 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.693695    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6"
Nov 03 00:03:02.715343 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.715311    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d36c485f-2e64-48e6-ae55-df84f85dd47b-kube-api-access-ndjvc\" (UniqueName: \"kubernetes.io/projected/d36c485f-2e64-48e6-ae55-df84f85dd47b-kube-api-access-ndjvc\") pod \"d36c485f-2e64-48e6-ae55-df84f85dd47b\" (UID: \"d36c485f-2e64-48e6-ae55-df84f85dd47b\") "
Nov 03 00:03:02.715599 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.715509    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d36c485f-2e64-48e6-ae55-df84f85dd47b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d36c485f-2e64-48e6-ae55-df84f85dd47b-plugins-config-volume\") pod \"d36c485f-2e64-48e6-ae55-df84f85dd47b\" (UID: \"d36c485f-2e64-48e6-ae55-df84f85dd47b\") "
Nov 03 00:03:02.716493 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.716463    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d36c485f-2e64-48e6-ae55-df84f85dd47b-plugins-config-volume" pod "d36c485f-2e64-48e6-ae55-df84f85dd47b" (UID: "d36c485f-2e64-48e6-ae55-df84f85dd47b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:03:02.719162 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.719127    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36c485f-2e64-48e6-ae55-df84f85dd47b-kube-api-access-ndjvc" pod "d36c485f-2e64-48e6-ae55-df84f85dd47b" (UID: "d36c485f-2e64-48e6-ae55-df84f85dd47b"). InnerVolumeSpecName "kube-api-access-ndjvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:03:02.816992 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.816899    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb92a5d9-abf6-4d20-9919-5edce78d957f-kube-api-access-2qqnn\" (UniqueName: \"kubernetes.io/projected/bb92a5d9-abf6-4d20-9919-5edce78d957f-kube-api-access-2qqnn\") pod \"bb92a5d9-abf6-4d20-9919-5edce78d957f\" (UID: \"bb92a5d9-abf6-4d20-9919-5edce78d957f\") "
Nov 03 00:03:02.817178 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.817005    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb92a5d9-abf6-4d20-9919-5edce78d957f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb92a5d9-abf6-4d20-9919-5edce78d957f-plugins-config-volume\") pod \"bb92a5d9-abf6-4d20-9919-5edce78d957f\" (UID: \"bb92a5d9-abf6-4d20-9919-5edce78d957f\") "
Nov 03 00:03:02.817451 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.817418    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb92a5d9-abf6-4d20-9919-5edce78d957f-plugins-config-volume" pod "bb92a5d9-abf6-4d20-9919-5edce78d957f" (UID: "bb92a5d9-abf6-4d20-9919-5edce78d957f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:03:02.817754 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.817737    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d36c485f-2e64-48e6-ae55-df84f85dd47b-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:03:02.817754 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.817757    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb92a5d9-abf6-4d20-9919-5edce78d957f-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:03:02.817877 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.817768    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndjvc\" (UniqueName: \"kubernetes.io/projected/d36c485f-2e64-48e6-ae55-df84f85dd47b-kube-api-access-ndjvc\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:03:02.819226 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.819198    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb92a5d9-abf6-4d20-9919-5edce78d957f-kube-api-access-2qqnn" pod "bb92a5d9-abf6-4d20-9919-5edce78d957f" (UID: "bb92a5d9-abf6-4d20-9919-5edce78d957f"). InnerVolumeSpecName "kube-api-access-2qqnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:03:02.878496 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.878462    4294 generic.go:356] "Generic (PLEG): container finished" podID="bb92a5d9-abf6-4d20-9919-5edce78d957f" containerID="05c49aa21512b56e110ff6513234ebfe9026482b3d3c875d5cde1cf9edfda65b" exitCode=0
Nov 03 00:03:02.878692 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.878516    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6"
Nov 03 00:03:02.878692 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.878546    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6" event={"ID":"bb92a5d9-abf6-4d20-9919-5edce78d957f","Type":"ContainerDied","Data":"05c49aa21512b56e110ff6513234ebfe9026482b3d3c875d5cde1cf9edfda65b"}
Nov 03 00:03:02.878692 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.878599    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6" event={"ID":"bb92a5d9-abf6-4d20-9919-5edce78d957f","Type":"ContainerDied","Data":"0f35a027358f568c94787225c77475da971000d6599fc65fef245e7aea8c75df"}
Nov 03 00:03:02.878692 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.878621    4294 scope.go:122] "RemoveContainer" containerID="05c49aa21512b56e110ff6513234ebfe9026482b3d3c875d5cde1cf9edfda65b"
Nov 03 00:03:02.880437 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.880412    4294 generic.go:356] "Generic (PLEG): container finished" podID="d36c485f-2e64-48e6-ae55-df84f85dd47b" containerID="83ff5d2d98bdddd2b77e21c42c75b6861ad91e3c270563ace06302b936b27593" exitCode=0
Nov 03 00:03:02.880577 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.880456    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj"
Nov 03 00:03:02.880577 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.880450    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj" event={"ID":"d36c485f-2e64-48e6-ae55-df84f85dd47b","Type":"ContainerDied","Data":"83ff5d2d98bdddd2b77e21c42c75b6861ad91e3c270563ace06302b936b27593"}
Nov 03 00:03:02.880687 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.880599    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj" event={"ID":"d36c485f-2e64-48e6-ae55-df84f85dd47b","Type":"ContainerDied","Data":"26c86a6c72b7a33523fb5168dfb1976bc11ba2e3cd57e6737a9e687b88a57ae3"}
Nov 03 00:03:02.890338 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.890315    4294 scope.go:122] "RemoveContainer" containerID="05c49aa21512b56e110ff6513234ebfe9026482b3d3c875d5cde1cf9edfda65b"
Nov 03 00:03:02.890790 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:03:02.890763    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05c49aa21512b56e110ff6513234ebfe9026482b3d3c875d5cde1cf9edfda65b\": container with ID starting with 05c49aa21512b56e110ff6513234ebfe9026482b3d3c875d5cde1cf9edfda65b not found: ID does not exist" containerID="05c49aa21512b56e110ff6513234ebfe9026482b3d3c875d5cde1cf9edfda65b"
Nov 03 00:03:02.890909 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.890802    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c49aa21512b56e110ff6513234ebfe9026482b3d3c875d5cde1cf9edfda65b"} err="failed to get container status \"05c49aa21512b56e110ff6513234ebfe9026482b3d3c875d5cde1cf9edfda65b\": rpc error: code = NotFound desc = could not find container \"05c49aa21512b56e110ff6513234ebfe9026482b3d3c875d5cde1cf9edfda65b\": container with ID starting with 05c49aa21512b56e110ff6513234ebfe9026482b3d3c875d5cde1cf9edfda65b not found: ID does not exist"
Nov 03 00:03:02.890909 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.890828    4294 scope.go:122] "RemoveContainer" containerID="83ff5d2d98bdddd2b77e21c42c75b6861ad91e3c270563ace06302b936b27593"
Nov 03 00:03:02.900665 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.900646    4294 scope.go:122] "RemoveContainer" containerID="83ff5d2d98bdddd2b77e21c42c75b6861ad91e3c270563ace06302b936b27593"
Nov 03 00:03:02.900910 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:03:02.900890    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ff5d2d98bdddd2b77e21c42c75b6861ad91e3c270563ace06302b936b27593\": container with ID starting with 83ff5d2d98bdddd2b77e21c42c75b6861ad91e3c270563ace06302b936b27593 not found: ID does not exist" containerID="83ff5d2d98bdddd2b77e21c42c75b6861ad91e3c270563ace06302b936b27593"
Nov 03 00:03:02.900969 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.900918    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ff5d2d98bdddd2b77e21c42c75b6861ad91e3c270563ace06302b936b27593"} err="failed to get container status \"83ff5d2d98bdddd2b77e21c42c75b6861ad91e3c270563ace06302b936b27593\": rpc error: code = NotFound desc = could not find container \"83ff5d2d98bdddd2b77e21c42c75b6861ad91e3c270563ace06302b936b27593\": container with ID starting with 83ff5d2d98bdddd2b77e21c42c75b6861ad91e3c270563ace06302b936b27593 not found: ID does not exist"
Nov 03 00:03:02.918872 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:02.918844    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qqnn\" (UniqueName: \"kubernetes.io/projected/bb92a5d9-abf6-4d20-9919-5edce78d957f-kube-api-access-2qqnn\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:03:03.040564 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:03.040528    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj"]
Nov 03 00:03:03.043957 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:03.043935    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj"]
Nov 03 00:03:03.145934 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:03.145851    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6"]
Nov 03 00:03:03.151919 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:03.151891    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f29e6-64b64f6f95-zkmn6"]
Nov 03 00:03:03.307784 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:03.307741    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1f29e6-6bcd4f6c-tq7tj" podUID="d36c485f-2e64-48e6-ae55-df84f85dd47b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.185:50052\" within 1s: context deadline exceeded"
Nov 03 00:03:04.549314 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:04.549281    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb92a5d9-abf6-4d20-9919-5edce78d957f" path="/var/lib/kubelet/pods/bb92a5d9-abf6-4d20-9919-5edce78d957f/volumes"
Nov 03 00:03:04.549777 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:04.549759    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d36c485f-2e64-48e6-ae55-df84f85dd47b" path="/var/lib/kubelet/pods/d36c485f-2e64-48e6-ae55-df84f85dd47b/volumes"
Nov 03 00:03:05.939126 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:05.939092    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:10.462195 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:10.462152    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:11.542010 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:11.541957    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:03:11.752681 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:03:11.752644    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:03:15.004874 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:15.004838    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:19.549834 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:19.549789    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:21.541994 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:21.541950    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:03:24.170935 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:24.170896    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:28.816217 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:28.816176    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:30.545797 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:30.545754    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:03:33.342440 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:33.342395    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:33.941735 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:33.941700    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:38.162913 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:38.162871    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:40.075568 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:40.075530    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:40.511382 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:40.511343    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2"]
Nov 03 00:03:40.545739 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:40.545707    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:03:40.673275 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:40.673239    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9"]
Nov 03 00:03:40.917033 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:40.916917    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:41.083663 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:41.083632    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:41.753922 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:03:41.753871    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:03:43.007318 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:43.007281    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:45.511861 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:45.511822    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2" podUID="37c6298e-b0e8-4f3f-80d7-62a04699cbc5" containerName="payload-processing" containerID="cri-o://9a1129f366ab8cbcd05a168945f74feab62cdff928977fdbdf7158b87ea4610a" gracePeriod=25
Nov 03 00:03:45.597199 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:45.597165    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:45.674057 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:45.674017    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9" podUID="0bf1c293-845f-4179-a74e-8203f5d07479" containerName="payload-pre-processing" containerID="cri-o://d9c52b80ce164d0db4f68710a021c6056ccc7c1c1dca909ba5e437319d3aaca3" gracePeriod=25
Nov 03 00:03:45.767032 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:45.766951    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:45.808638 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:45.808611    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2"
Nov 03 00:03:45.913239 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:45.913204    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37c6298e-b0e8-4f3f-80d7-62a04699cbc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37c6298e-b0e8-4f3f-80d7-62a04699cbc5-plugins-config-volume\") pod \"37c6298e-b0e8-4f3f-80d7-62a04699cbc5\" (UID: \"37c6298e-b0e8-4f3f-80d7-62a04699cbc5\") "
Nov 03 00:03:45.913528 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:45.913503    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c6298e-b0e8-4f3f-80d7-62a04699cbc5-kube-api-access-gxc89\" (UniqueName: \"kubernetes.io/projected/37c6298e-b0e8-4f3f-80d7-62a04699cbc5-kube-api-access-gxc89\") pod \"37c6298e-b0e8-4f3f-80d7-62a04699cbc5\" (UID: \"37c6298e-b0e8-4f3f-80d7-62a04699cbc5\") "
Nov 03 00:03:45.913650 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:45.913582    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37c6298e-b0e8-4f3f-80d7-62a04699cbc5-plugins-config-volume" pod "37c6298e-b0e8-4f3f-80d7-62a04699cbc5" (UID: "37c6298e-b0e8-4f3f-80d7-62a04699cbc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:03:45.914242 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:45.914218    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37c6298e-b0e8-4f3f-80d7-62a04699cbc5-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:03:45.915721 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:45.915694    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c6298e-b0e8-4f3f-80d7-62a04699cbc5-kube-api-access-gxc89" pod "37c6298e-b0e8-4f3f-80d7-62a04699cbc5" (UID: "37c6298e-b0e8-4f3f-80d7-62a04699cbc5"). InnerVolumeSpecName "kube-api-access-gxc89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:03:45.927889 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:45.927865    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9"
Nov 03 00:03:46.015619 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.015582    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxc89\" (UniqueName: \"kubernetes.io/projected/37c6298e-b0e8-4f3f-80d7-62a04699cbc5-kube-api-access-gxc89\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:03:46.116304 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.116228    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bf1c293-845f-4179-a74e-8203f5d07479-kube-api-access-msrbm\" (UniqueName: \"kubernetes.io/projected/0bf1c293-845f-4179-a74e-8203f5d07479-kube-api-access-msrbm\") pod \"0bf1c293-845f-4179-a74e-8203f5d07479\" (UID: \"0bf1c293-845f-4179-a74e-8203f5d07479\") "
Nov 03 00:03:46.116461 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.116311    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bf1c293-845f-4179-a74e-8203f5d07479-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf1c293-845f-4179-a74e-8203f5d07479-plugins-config-volume\") pod \"0bf1c293-845f-4179-a74e-8203f5d07479\" (UID: \"0bf1c293-845f-4179-a74e-8203f5d07479\") "
Nov 03 00:03:46.116682 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.116652    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf1c293-845f-4179-a74e-8203f5d07479-plugins-config-volume" pod "0bf1c293-845f-4179-a74e-8203f5d07479" (UID: "0bf1c293-845f-4179-a74e-8203f5d07479"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:03:46.116842 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.116824    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf1c293-845f-4179-a74e-8203f5d07479-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:03:46.118243 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.118223    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf1c293-845f-4179-a74e-8203f5d07479-kube-api-access-msrbm" pod "0bf1c293-845f-4179-a74e-8203f5d07479" (UID: "0bf1c293-845f-4179-a74e-8203f5d07479"). InnerVolumeSpecName "kube-api-access-msrbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:03:46.179498 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.179462    4294 generic.go:356] "Generic (PLEG): container finished" podID="0bf1c293-845f-4179-a74e-8203f5d07479" containerID="d9c52b80ce164d0db4f68710a021c6056ccc7c1c1dca909ba5e437319d3aaca3" exitCode=0
Nov 03 00:03:46.179639 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.179522    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9"
Nov 03 00:03:46.179639 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.179545    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9" event={"ID":"0bf1c293-845f-4179-a74e-8203f5d07479","Type":"ContainerDied","Data":"d9c52b80ce164d0db4f68710a021c6056ccc7c1c1dca909ba5e437319d3aaca3"}
Nov 03 00:03:46.179639 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.179602    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9" event={"ID":"0bf1c293-845f-4179-a74e-8203f5d07479","Type":"ContainerDied","Data":"8a31b5229b6702faf4b65acd27acd6ad09aa4b3af5055af479f157cf5c22ada3"}
Nov 03 00:03:46.179639 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.179625    4294 scope.go:122] "RemoveContainer" containerID="d9c52b80ce164d0db4f68710a021c6056ccc7c1c1dca909ba5e437319d3aaca3"
Nov 03 00:03:46.181157 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.181126    4294 generic.go:356] "Generic (PLEG): container finished" podID="37c6298e-b0e8-4f3f-80d7-62a04699cbc5" containerID="9a1129f366ab8cbcd05a168945f74feab62cdff928977fdbdf7158b87ea4610a" exitCode=0
Nov 03 00:03:46.181289 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.181173    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2" event={"ID":"37c6298e-b0e8-4f3f-80d7-62a04699cbc5","Type":"ContainerDied","Data":"9a1129f366ab8cbcd05a168945f74feab62cdff928977fdbdf7158b87ea4610a"}
Nov 03 00:03:46.181289 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.181177    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2"
Nov 03 00:03:46.181289 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.181207    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2" event={"ID":"37c6298e-b0e8-4f3f-80d7-62a04699cbc5","Type":"ContainerDied","Data":"ce7e56a4e5e7e23e74df41d63074389442fa666be796140e9f761e728dab7a44"}
Nov 03 00:03:46.194300 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.194277    4294 scope.go:122] "RemoveContainer" containerID="d9c52b80ce164d0db4f68710a021c6056ccc7c1c1dca909ba5e437319d3aaca3"
Nov 03 00:03:46.194590 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:03:46.194571    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9c52b80ce164d0db4f68710a021c6056ccc7c1c1dca909ba5e437319d3aaca3\": container with ID starting with d9c52b80ce164d0db4f68710a021c6056ccc7c1c1dca909ba5e437319d3aaca3 not found: ID does not exist" containerID="d9c52b80ce164d0db4f68710a021c6056ccc7c1c1dca909ba5e437319d3aaca3"
Nov 03 00:03:46.194644 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.194598    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9c52b80ce164d0db4f68710a021c6056ccc7c1c1dca909ba5e437319d3aaca3"} err="failed to get container status \"d9c52b80ce164d0db4f68710a021c6056ccc7c1c1dca909ba5e437319d3aaca3\": rpc error: code = NotFound desc = could not find container \"d9c52b80ce164d0db4f68710a021c6056ccc7c1c1dca909ba5e437319d3aaca3\": container with ID starting with d9c52b80ce164d0db4f68710a021c6056ccc7c1c1dca909ba5e437319d3aaca3 not found: ID does not exist"
Nov 03 00:03:46.194644 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.194617    4294 scope.go:122] "RemoveContainer" containerID="9a1129f366ab8cbcd05a168945f74feab62cdff928977fdbdf7158b87ea4610a"
Nov 03 00:03:46.203524 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.203509    4294 scope.go:122] "RemoveContainer" containerID="9a1129f366ab8cbcd05a168945f74feab62cdff928977fdbdf7158b87ea4610a"
Nov 03 00:03:46.203759 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:03:46.203741    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1129f366ab8cbcd05a168945f74feab62cdff928977fdbdf7158b87ea4610a\": container with ID starting with 9a1129f366ab8cbcd05a168945f74feab62cdff928977fdbdf7158b87ea4610a not found: ID does not exist" containerID="9a1129f366ab8cbcd05a168945f74feab62cdff928977fdbdf7158b87ea4610a"
Nov 03 00:03:46.203814 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.203766    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1129f366ab8cbcd05a168945f74feab62cdff928977fdbdf7158b87ea4610a"} err="failed to get container status \"9a1129f366ab8cbcd05a168945f74feab62cdff928977fdbdf7158b87ea4610a\": rpc error: code = NotFound desc = could not find container \"9a1129f366ab8cbcd05a168945f74feab62cdff928977fdbdf7158b87ea4610a\": container with ID starting with 9a1129f366ab8cbcd05a168945f74feab62cdff928977fdbdf7158b87ea4610a not found: ID does not exist"
Nov 03 00:03:46.218680 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.218657    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msrbm\" (UniqueName: \"kubernetes.io/projected/0bf1c293-845f-4179-a74e-8203f5d07479-kube-api-access-msrbm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:03:46.339122 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.339073    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2"]
Nov 03 00:03:46.342277 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.342256    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ad18ae-74f59bdf95-ndqb2"]
Nov 03 00:03:46.447002 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.446898    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9"]
Nov 03 00:03:46.456146 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.456105    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ad18ae-7958c4cbb7-9zbv9"]
Nov 03 00:03:46.549133 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.549103    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bf1c293-845f-4179-a74e-8203f5d07479" path="/var/lib/kubelet/pods/0bf1c293-845f-4179-a74e-8203f5d07479/volumes"
Nov 03 00:03:46.549519 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:46.549506    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37c6298e-b0e8-4f3f-80d7-62a04699cbc5" path="/var/lib/kubelet/pods/37c6298e-b0e8-4f3f-80d7-62a04699cbc5/volumes"
Nov 03 00:03:47.233395 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:47.233353    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:47.639257 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:47.639225    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:50.546896 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:50.546862    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:03:52.273577 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:52.273545    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:53.386859 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:53.386823    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:54.351852 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:54.351811    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:54.535096 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:54.535061    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:57.139239 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:57.139208    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:57.422638 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:57.422543    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:57.584667 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:57.584631    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:03:59.259693 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:03:59.259655    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:00.545959 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:00.545918    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:04:02.262947 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:02.262916    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:06.849805 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:06.849764    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:10.545627 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:10.545591    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:04:11.479821 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:11.479784    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:11.752899 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:04:11.752865    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:04:16.005124 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:16.005087    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:19.897495 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:19.897460    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"]
Nov 03 00:04:19.897948 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:19.897650    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv" podUID="7ffb04c4-4f77-4da0-b266-b72574803dd0" containerName="maas-api" containerID="cri-o://1b428996af04f205336b268bc39c445a1e93a6e48a6616b0a7f02ccf06be5bbd" gracePeriod=30
Nov 03 00:04:20.036466 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.036435    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:20.049008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.048947    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv" podUID="7ffb04c4-4f77-4da0-b266-b72574803dd0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.196:8443/health\": dial tcp 10.133.0.196:8443: connect: connection refused"
Nov 03 00:04:20.173463 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.173439    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"
Nov 03 00:04:20.256480 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.256453    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ffb04c4-4f77-4da0-b266-b72574803dd0-kube-api-access-rzgtz\" (UniqueName: \"kubernetes.io/projected/7ffb04c4-4f77-4da0-b266-b72574803dd0-kube-api-access-rzgtz\") pod \"7ffb04c4-4f77-4da0-b266-b72574803dd0\" (UID: \"7ffb04c4-4f77-4da0-b266-b72574803dd0\") "
Nov 03 00:04:20.256634 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.256595    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ffb04c4-4f77-4da0-b266-b72574803dd0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ffb04c4-4f77-4da0-b266-b72574803dd0-metrics-certs\") pod \"7ffb04c4-4f77-4da0-b266-b72574803dd0\" (UID: \"7ffb04c4-4f77-4da0-b266-b72574803dd0\") "
Nov 03 00:04:20.256677 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.256660    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ffb04c4-4f77-4da0-b266-b72574803dd0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ffb04c4-4f77-4da0-b266-b72574803dd0-maas-api-tls\") pod \"7ffb04c4-4f77-4da0-b266-b72574803dd0\" (UID: \"7ffb04c4-4f77-4da0-b266-b72574803dd0\") "
Nov 03 00:04:20.256814 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.256791    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ffb04c4-4f77-4da0-b266-b72574803dd0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ffb04c4-4f77-4da0-b266-b72574803dd0-openshift-service-ca\") pod \"7ffb04c4-4f77-4da0-b266-b72574803dd0\" (UID: \"7ffb04c4-4f77-4da0-b266-b72574803dd0\") "
Nov 03 00:04:20.257215 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.257184    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ffb04c4-4f77-4da0-b266-b72574803dd0-openshift-service-ca" pod "7ffb04c4-4f77-4da0-b266-b72574803dd0" (UID: "7ffb04c4-4f77-4da0-b266-b72574803dd0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:04:20.257445 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.257421    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ffb04c4-4f77-4da0-b266-b72574803dd0-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:04:20.258626 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.258602    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ffb04c4-4f77-4da0-b266-b72574803dd0-kube-api-access-rzgtz" pod "7ffb04c4-4f77-4da0-b266-b72574803dd0" (UID: "7ffb04c4-4f77-4da0-b266-b72574803dd0"). InnerVolumeSpecName "kube-api-access-rzgtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:04:20.258822 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.258801    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ffb04c4-4f77-4da0-b266-b72574803dd0-metrics-certs" pod "7ffb04c4-4f77-4da0-b266-b72574803dd0" (UID: "7ffb04c4-4f77-4da0-b266-b72574803dd0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:04:20.259013 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.258972    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ffb04c4-4f77-4da0-b266-b72574803dd0-maas-api-tls" pod "7ffb04c4-4f77-4da0-b266-b72574803dd0" (UID: "7ffb04c4-4f77-4da0-b266-b72574803dd0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:04:20.358664 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.358632    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ffb04c4-4f77-4da0-b266-b72574803dd0-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:04:20.358664 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.358659    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzgtz\" (UniqueName: \"kubernetes.io/projected/7ffb04c4-4f77-4da0-b266-b72574803dd0-kube-api-access-rzgtz\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:04:20.358664 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.358670    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ffb04c4-4f77-4da0-b266-b72574803dd0-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:04:20.423922 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.423837    4294 generic.go:356] "Generic (PLEG): container finished" podID="7ffb04c4-4f77-4da0-b266-b72574803dd0" containerID="1b428996af04f205336b268bc39c445a1e93a6e48a6616b0a7f02ccf06be5bbd" exitCode=0
Nov 03 00:04:20.423922 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.423875    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"
Nov 03 00:04:20.424171 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.423911    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv" event={"ID":"7ffb04c4-4f77-4da0-b266-b72574803dd0","Type":"ContainerDied","Data":"1b428996af04f205336b268bc39c445a1e93a6e48a6616b0a7f02ccf06be5bbd"}
Nov 03 00:04:20.424171 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.423972    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv" event={"ID":"7ffb04c4-4f77-4da0-b266-b72574803dd0","Type":"ContainerDied","Data":"05adc7c163dab074086beff5d15df0e4c0850f1a577884e3244a2e2e68112aa2"}
Nov 03 00:04:20.424171 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.424010    4294 scope.go:122] "RemoveContainer" containerID="1b428996af04f205336b268bc39c445a1e93a6e48a6616b0a7f02ccf06be5bbd"
Nov 03 00:04:20.436422 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.436396    4294 scope.go:122] "RemoveContainer" containerID="1b428996af04f205336b268bc39c445a1e93a6e48a6616b0a7f02ccf06be5bbd"
Nov 03 00:04:20.436713 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:04:20.436686    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b428996af04f205336b268bc39c445a1e93a6e48a6616b0a7f02ccf06be5bbd\": container with ID starting with 1b428996af04f205336b268bc39c445a1e93a6e48a6616b0a7f02ccf06be5bbd not found: ID does not exist" containerID="1b428996af04f205336b268bc39c445a1e93a6e48a6616b0a7f02ccf06be5bbd"
Nov 03 00:04:20.436792 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.436723    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b428996af04f205336b268bc39c445a1e93a6e48a6616b0a7f02ccf06be5bbd"} err="failed to get container status \"1b428996af04f205336b268bc39c445a1e93a6e48a6616b0a7f02ccf06be5bbd\": rpc error: code = NotFound desc = could not find container \"1b428996af04f205336b268bc39c445a1e93a6e48a6616b0a7f02ccf06be5bbd\": container with ID starting with 1b428996af04f205336b268bc39c445a1e93a6e48a6616b0a7f02ccf06be5bbd not found: ID does not exist"
Nov 03 00:04:20.545884 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.545847    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:04:20.582731 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.582702    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"]
Nov 03 00:04:20.587855 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:20.587830    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1f29e6-f7c677578-4lzkv"]
Nov 03 00:04:21.845932 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:21.845888    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:22.023188 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:22.023141    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:22.431919 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:22.431887    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:22.562014 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:22.559797    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ffb04c4-4f77-4da0-b266-b72574803dd0" path="/var/lib/kubelet/pods/7ffb04c4-4f77-4da0-b266-b72574803dd0/volumes"
Nov 03 00:04:25.029476 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:25.029447    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:25.216849 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:25.216797    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:26.644409 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:26.644375    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:26.850635 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:26.850606    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:27.035098 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:27.035057    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:30.545857 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:30.545816    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:04:32.200733 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:32.200695    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:36.843507 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:36.843469    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:40.551219 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:40.551176    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:04:41.414110 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:41.414075    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:41.750432 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:04:41.750398    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:04:45.542271 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:45.542240    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:04:45.977047 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:45.976950    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:46.200330 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:46.200299    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:50.120305 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:50.120271    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:52.101335 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:52.101296    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:54.286428 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:54.286393    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:54.470841 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:54.470802    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:55.565102 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:55.565063    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:04:55.977734 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:55.977638    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:56.336003 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:56.313528    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:56.347551 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:56.344081    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-759489599d-ft2jp"]
Nov 03 00:04:56.350126 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:56.350095    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:04:56.476751 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:56.476711    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-64887bb858-xh446"]
Nov 03 00:04:56.736061 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:56.701938    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:56.762004 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:56.757893    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf" podUID="2f110c17-1134-4f5c-b9a6-51abb2ac481f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:04:56.908998 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:56.901394    4294 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf" podUID="2f110c17-1134-4f5c-b9a6-51abb2ac481f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:04:56.966014 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:56.965959    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:57.019193 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:57.019150    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:04:57.020225 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:57.020186    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9" podUID="43e3d0d6-1a8e-4536-bc46-b5785124405a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:04:57.074123 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:57.065419    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 00:04:57.341595 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:57.341503    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:57.587743 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:04:57.587704    4294 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7ea550d38410: secret "agc-cred-7ea550d38410" not found
Nov 03 00:04:57.587743 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:04:57.587736    4294 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a610cb5fdd81: secret "agc-cred-a610cb5fdd81" not found
Nov 03 00:04:57.587743 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:04:57.587745    4294 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-eb3055757321: secret "agc-cred-eb3055757321" not found
Nov 03 00:04:57.588098 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:04:57.587758    4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-64887bb858-xh446: [secret "agc-cred-7ea550d38410" not found, secret "agc-cred-a610cb5fdd81" not found, secret "agc-cred-eb3055757321" not found]
Nov 03 00:04:57.588098 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:04:57.587837    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials podName:6ae49deb-156f-4f33-a81d-06bc93b7dcb8 nodeName:}" failed. No retries permitted until 2026-11-03 00:06:59.587818001 +0000 UTC m=+4459.156172779 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials") pod "payload-processing-external-model-64887bb858-xh446" (UID: "6ae49deb-156f-4f33-a81d-06bc93b7dcb8") : [secret "agc-cred-7ea550d38410" not found, secret "agc-cred-a610cb5fdd81" not found, secret "agc-cred-eb3055757321" not found]
Nov 03 00:04:57.664478 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:57.664400    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:58.694539 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:58.694499    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:04:58.976169 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:04:58.976078    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:01.348285 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:01.348242    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing" containerID="cri-o://1b49f11572f20432a37be02bbea357167c7e53eee93fc85e17ea5f8576287c2d" gracePeriod=25
Nov 03 00:05:01.477875 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:01.477841    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446" podUID="6ae49deb-156f-4f33-a81d-06bc93b7dcb8" containerName="payload-processing" containerID="cri-o://9141eb06037c5b06f23328268a22d117fa164ed604503e0485e50ed0cacb49a8" gracePeriod=25
Nov 03 00:05:01.961816 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:01.961782    4294 generic.go:356] "Generic (PLEG): container finished" podID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerID="1b49f11572f20432a37be02bbea357167c7e53eee93fc85e17ea5f8576287c2d" exitCode=0
Nov 03 00:05:01.962217 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:01.962034    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" event={"ID":"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593","Type":"ContainerDied","Data":"1b49f11572f20432a37be02bbea357167c7e53eee93fc85e17ea5f8576287c2d"}
Nov 03 00:05:01.964320 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:01.964296    4294 generic.go:356] "Generic (PLEG): container finished" podID="6ae49deb-156f-4f33-a81d-06bc93b7dcb8" containerID="9141eb06037c5b06f23328268a22d117fa164ed604503e0485e50ed0cacb49a8" exitCode=0
Nov 03 00:05:01.964593 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:01.964576    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446" event={"ID":"6ae49deb-156f-4f33-a81d-06bc93b7dcb8","Type":"ContainerDied","Data":"9141eb06037c5b06f23328268a22d117fa164ed604503e0485e50ed0cacb49a8"}
Nov 03 00:05:02.034000 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.022466    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp"
Nov 03 00:05:02.064001 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.063165    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446"
Nov 03 00:05:02.101455 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.101422    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-provider-credentials\") pod \"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593\" (UID: \"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593\") "
Nov 03 00:05:02.101713 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.101697    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-kube-api-access-r79jc\" (UniqueName: \"kubernetes.io/projected/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-kube-api-access-r79jc\") pod \"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593\" (UID: \"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593\") "
Nov 03 00:05:02.101853 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.101838    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials\") pod \"6ae49deb-156f-4f33-a81d-06bc93b7dcb8\" (UID: \"6ae49deb-156f-4f33-a81d-06bc93b7dcb8\") "
Nov 03 00:05:02.102015 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.101999    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-plugins-config-volume\") pod \"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593\" (UID: \"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593\") "
Nov 03 00:05:02.102161 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.102145    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-plugins-config-volume\") pod \"6ae49deb-156f-4f33-a81d-06bc93b7dcb8\" (UID: \"6ae49deb-156f-4f33-a81d-06bc93b7dcb8\") "
Nov 03 00:05:02.102336 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.102321    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-routing-overlay-volume\") pod \"6ae49deb-156f-4f33-a81d-06bc93b7dcb8\" (UID: \"6ae49deb-156f-4f33-a81d-06bc93b7dcb8\") "
Nov 03 00:05:02.102474 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.102460    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-routing-overlay-volume\") pod \"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593\" (UID: \"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593\") "
Nov 03 00:05:02.102610 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.102597    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-kube-api-access-z4rdq\" (UniqueName: \"kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-kube-api-access-z4rdq\") pod \"6ae49deb-156f-4f33-a81d-06bc93b7dcb8\" (UID: \"6ae49deb-156f-4f33-a81d-06bc93b7dcb8\") "
Nov 03 00:05:02.103941 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.103919    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-plugins-config-volume" pod "ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" (UID: "ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:05:02.106836 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.105359    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-plugins-config-volume" pod "6ae49deb-156f-4f33-a81d-06bc93b7dcb8" (UID: "6ae49deb-156f-4f33-a81d-06bc93b7dcb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:05:02.106836 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.106661    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-routing-overlay-volume" pod "ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" (UID: "ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:05:02.106836 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.106811    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-routing-overlay-volume" pod "6ae49deb-156f-4f33-a81d-06bc93b7dcb8" (UID: "6ae49deb-156f-4f33-a81d-06bc93b7dcb8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:05:02.112818 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.112785    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials" pod "6ae49deb-156f-4f33-a81d-06bc93b7dcb8" (UID: "6ae49deb-156f-4f33-a81d-06bc93b7dcb8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:05:02.113122 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.113089    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-kube-api-access-r79jc" pod "ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" (UID: "ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593"). InnerVolumeSpecName "kube-api-access-r79jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:05:02.151513 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.150159    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-kube-api-access-z4rdq" pod "6ae49deb-156f-4f33-a81d-06bc93b7dcb8" (UID: "6ae49deb-156f-4f33-a81d-06bc93b7dcb8"). InnerVolumeSpecName "kube-api-access-z4rdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:05:02.170313 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.170181    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-provider-credentials" pod "ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" (UID: "ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:05:02.205084 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.205055    4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-routing-overlay-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:02.205084 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.205081    4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-routing-overlay-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:02.205285 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.205093    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4rdq\" (UniqueName: \"kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-kube-api-access-z4rdq\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:02.205285 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.205102    4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-provider-credentials\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:02.205285 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.205111    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r79jc\" (UniqueName: \"kubernetes.io/projected/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-kube-api-access-r79jc\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:02.205285 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.205119    4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-provider-credentials\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:02.205285 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.205129    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:02.205285 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.205140    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae49deb-156f-4f33-a81d-06bc93b7dcb8-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:02.510677 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.510633    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:02.972241 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.972171    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp"
Nov 03 00:05:02.972241 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.972185    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-759489599d-ft2jp" event={"ID":"ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593","Type":"ContainerDied","Data":"04fb8408f602d738081cdb3e270a14df4e5ed626e27ba6cf8433f48367936f89"}
Nov 03 00:05:02.972501 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.972252    4294 scope.go:122] "RemoveContainer" containerID="1b49f11572f20432a37be02bbea357167c7e53eee93fc85e17ea5f8576287c2d"
Nov 03 00:05:02.973832 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.973811    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446"
Nov 03 00:05:02.973916 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.973853    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64887bb858-xh446" event={"ID":"6ae49deb-156f-4f33-a81d-06bc93b7dcb8","Type":"ContainerDied","Data":"7ba9cb4a44a600905ee143ac134a35a645241f3ded3f90ea9a02b35e031a8b3c"}
Nov 03 00:05:02.983063 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.983044    4294 scope.go:122] "RemoveContainer" containerID="9141eb06037c5b06f23328268a22d117fa164ed604503e0485e50ed0cacb49a8"
Nov 03 00:05:02.994502 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.994481    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"]
Nov 03 00:05:02.995133 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995118    4294 cpu_manager.go:414] "containerMap: removing container" podUID="37c6298e-b0e8-4f3f-80d7-62a04699cbc5" containerName="payload-processing"
Nov 03 00:05:02.995187 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995134    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c6298e-b0e8-4f3f-80d7-62a04699cbc5" containerName="payload-processing"
Nov 03 00:05:02.995187 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995150    4294 cpu_manager.go:414] "containerMap: removing container" podUID="bb92a5d9-abf6-4d20-9919-5edce78d957f" containerName="payload-pre-processing"
Nov 03 00:05:02.995187 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995156    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb92a5d9-abf6-4d20-9919-5edce78d957f" containerName="payload-pre-processing"
Nov 03 00:05:02.995187 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995176    4294 cpu_manager.go:414] "containerMap: removing container" podUID="d36c485f-2e64-48e6-ae55-df84f85dd47b" containerName="payload-processing"
Nov 03 00:05:02.995187 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995181    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d36c485f-2e64-48e6-ae55-df84f85dd47b" containerName="payload-processing"
Nov 03 00:05:02.995362 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995191    4294 cpu_manager.go:414] "containerMap: removing container" podUID="6ae49deb-156f-4f33-a81d-06bc93b7dcb8" containerName="payload-processing"
Nov 03 00:05:02.995362 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995197    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae49deb-156f-4f33-a81d-06bc93b7dcb8" containerName="payload-processing"
Nov 03 00:05:02.995362 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995207    4294 cpu_manager.go:414] "containerMap: removing container" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing"
Nov 03 00:05:02.995362 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995212    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing"
Nov 03 00:05:02.995362 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995220    4294 cpu_manager.go:414] "containerMap: removing container" podUID="0bf1c293-845f-4179-a74e-8203f5d07479" containerName="payload-pre-processing"
Nov 03 00:05:02.995362 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995225    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bf1c293-845f-4179-a74e-8203f5d07479" containerName="payload-pre-processing"
Nov 03 00:05:02.995362 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995234    4294 cpu_manager.go:414] "containerMap: removing container" podUID="7ffb04c4-4f77-4da0-b266-b72574803dd0" containerName="maas-api"
Nov 03 00:05:02.995362 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995239    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ffb04c4-4f77-4da0-b266-b72574803dd0" containerName="maas-api"
Nov 03 00:05:02.995362 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995324    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ffb04c4-4f77-4da0-b266-b72574803dd0" containerName="maas-api"
Nov 03 00:05:02.995362 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995342    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c6298e-b0e8-4f3f-80d7-62a04699cbc5" containerName="payload-processing"
Nov 03 00:05:02.995362 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995349    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb92a5d9-abf6-4d20-9919-5edce78d957f" containerName="payload-pre-processing"
Nov 03 00:05:02.995362 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995360    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d36c485f-2e64-48e6-ae55-df84f85dd47b" containerName="payload-processing"
Nov 03 00:05:02.995741 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995368    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae49deb-156f-4f33-a81d-06bc93b7dcb8" containerName="payload-processing"
Nov 03 00:05:02.995741 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995375    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" containerName="payload-processing"
Nov 03 00:05:02.995741 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:02.995381    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bf1c293-845f-4179-a74e-8203f5d07479" containerName="payload-pre-processing"
Nov 03 00:05:03.000528 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.000510    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"
Nov 03 00:05:03.014942 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.014920    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j647q\" (UniqueName: \"kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-kube-api-access-j647q\") pod \"payload-processing-external-model-795b95959c-rpkc9\" (UID: \"596732d7-7a22-4c49-81c7-e76f5694077a\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"
Nov 03 00:05:03.015054 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.014953    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/596732d7-7a22-4c49-81c7-e76f5694077a-routing-overlay-volume\") pod \"payload-processing-external-model-795b95959c-rpkc9\" (UID: \"596732d7-7a22-4c49-81c7-e76f5694077a\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"
Nov 03 00:05:03.015127 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.015114    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/596732d7-7a22-4c49-81c7-e76f5694077a-plugins-config-volume\") pod \"payload-processing-external-model-795b95959c-rpkc9\" (UID: \"596732d7-7a22-4c49-81c7-e76f5694077a\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"
Nov 03 00:05:03.015188 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.015175    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-provider-credentials\") pod \"payload-processing-external-model-795b95959c-rpkc9\" (UID: \"596732d7-7a22-4c49-81c7-e76f5694077a\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"
Nov 03 00:05:03.051649 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.051624    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 00:05:03.051773 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.051684    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 00:05:03.051773 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.051745    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 00:05:03.051872 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.051809    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-9692b\""
Nov 03 00:05:03.064610 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.064575    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:03.076658 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.076634    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"]
Nov 03 00:05:03.116716 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.116686    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/596732d7-7a22-4c49-81c7-e76f5694077a-plugins-config-volume\") pod \"payload-processing-external-model-795b95959c-rpkc9\" (UID: \"596732d7-7a22-4c49-81c7-e76f5694077a\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"
Nov 03 00:05:03.116878 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.116749    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-provider-credentials\") pod \"payload-processing-external-model-795b95959c-rpkc9\" (UID: \"596732d7-7a22-4c49-81c7-e76f5694077a\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"
Nov 03 00:05:03.116878 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.116799    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j647q\" (UniqueName: \"kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-kube-api-access-j647q\") pod \"payload-processing-external-model-795b95959c-rpkc9\" (UID: \"596732d7-7a22-4c49-81c7-e76f5694077a\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"
Nov 03 00:05:03.116878 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.116818    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/596732d7-7a22-4c49-81c7-e76f5694077a-routing-overlay-volume\") pod \"payload-processing-external-model-795b95959c-rpkc9\" (UID: \"596732d7-7a22-4c49-81c7-e76f5694077a\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"
Nov 03 00:05:03.117370 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.117351    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/596732d7-7a22-4c49-81c7-e76f5694077a-routing-overlay-volume\") pod \"payload-processing-external-model-795b95959c-rpkc9\" (UID: \"596732d7-7a22-4c49-81c7-e76f5694077a\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"
Nov 03 00:05:03.117421 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.117407    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/596732d7-7a22-4c49-81c7-e76f5694077a-plugins-config-volume\") pod \"payload-processing-external-model-795b95959c-rpkc9\" (UID: \"596732d7-7a22-4c49-81c7-e76f5694077a\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"
Nov 03 00:05:03.135036 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.135006    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-759489599d-ft2jp"]
Nov 03 00:05:03.138468 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.138440    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-759489599d-ft2jp"]
Nov 03 00:05:03.152481 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.152437    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f2dbbb870d2c\""
Nov 03 00:05:03.152615 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.152553    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 00:05:03.159969 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.159943    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-provider-credentials\") pod \"payload-processing-external-model-795b95959c-rpkc9\" (UID: \"596732d7-7a22-4c49-81c7-e76f5694077a\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"
Nov 03 00:05:03.176103 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.176078    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j647q\" (UniqueName: \"kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-kube-api-access-j647q\") pod \"payload-processing-external-model-795b95959c-rpkc9\" (UID: \"596732d7-7a22-4c49-81c7-e76f5694077a\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"
Nov 03 00:05:03.314310 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.314285    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"
Nov 03 00:05:03.355293 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.355256    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-64887bb858-xh446"]
Nov 03 00:05:03.359653 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.359623    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-64887bb858-xh446"]
Nov 03 00:05:03.694740 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.694018    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"]
Nov 03 00:05:03.694740 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:05:03.694241    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596732d7_7a22_4c49_81c7_e76f5694077a.slice/crio-1e9bb07eae63a5b4a61741ceb6dcec984058e033ef98cc73ed431f1fe9243298 WatchSource:0}: Error finding container 1e9bb07eae63a5b4a61741ceb6dcec984058e033ef98cc73ed431f1fe9243298: Status 404 returned error can't find the container with id 1e9bb07eae63a5b4a61741ceb6dcec984058e033ef98cc73ed431f1fe9243298
Nov 03 00:05:03.981685 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.981584    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9" event={"ID":"596732d7-7a22-4c49-81c7-e76f5694077a","Type":"ContainerStarted","Data":"34a87cae71216f90c8d771a1ab36a06e117dc25cc2b3ee027c2a6c000a7b6a99"}
Nov 03 00:05:03.981685 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.981672    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9" event={"ID":"596732d7-7a22-4c49-81c7-e76f5694077a","Type":"ContainerStarted","Data":"1e9bb07eae63a5b4a61741ceb6dcec984058e033ef98cc73ed431f1fe9243298"}
Nov 03 00:05:03.981898 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:03.981791    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"
Nov 03 00:05:04.315231 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:04.315200    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:04.548554 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:04.548518    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae49deb-156f-4f33-a81d-06bc93b7dcb8" path="/var/lib/kubelet/pods/6ae49deb-156f-4f33-a81d-06bc93b7dcb8/volumes"
Nov 03 00:05:04.548972 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:04.548956    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593" path="/var/lib/kubelet/pods/ddc5dfb6-80a9-4f88-9ca0-e0f450e3a593/volumes"
Nov 03 00:05:04.814445 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:04.814394    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:04.996912 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:04.996852    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9" podStartSLOduration=2.996838191 podStartE2EDuration="2.996838191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:04.101487686 +0000 UTC m=+4343.669842473" watchObservedRunningTime="2026-11-03 00:05:04.996838191 +0000 UTC m=+4344.565192979"
Nov 03 00:05:04.998313 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:04.998289    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"]
Nov 03 00:05:05.016958 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:05.016933    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:05.056096 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:05:05.056049    4294 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f2dbbb870d2c: secret "agc-cred-f2dbbb870d2c" not found
Nov 03 00:05:05.056096 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:05:05.056095    4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-795b95959c-rpkc9: secret "agc-cred-f2dbbb870d2c" not found
Nov 03 00:05:05.056313 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:05:05.056175    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-provider-credentials podName:596732d7-7a22-4c49-81c7-e76f5694077a nodeName:}" failed. No retries permitted until 2026-11-03 00:05:05.556156052 +0000 UTC m=+4345.124510839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-provider-credentials") pod "payload-processing-external-model-795b95959c-rpkc9" (UID: "596732d7-7a22-4c49-81c7-e76f5694077a") : secret "agc-cred-f2dbbb870d2c" not found
Nov 03 00:05:05.201558 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:05.201468    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:05.562853 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:05:05.562805    4294 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f2dbbb870d2c: secret "agc-cred-f2dbbb870d2c" not found
Nov 03 00:05:05.562853 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:05:05.562840    4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-795b95959c-rpkc9: secret "agc-cred-f2dbbb870d2c" not found
Nov 03 00:05:05.563177 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:05:05.562899    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-provider-credentials podName:596732d7-7a22-4c49-81c7-e76f5694077a nodeName:}" failed. No retries permitted until 2026-11-03 00:05:06.562881013 +0000 UTC m=+4346.131235791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-provider-credentials") pod "payload-processing-external-model-795b95959c-rpkc9" (UID: "596732d7-7a22-4c49-81c7-e76f5694077a") : secret "agc-cred-f2dbbb870d2c" not found
Nov 03 00:05:06.585878 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:05:06.585851    4294 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f2dbbb870d2c: secret "agc-cred-f2dbbb870d2c" not found
Nov 03 00:05:06.585878 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:05:06.585875    4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-795b95959c-rpkc9: secret "agc-cred-f2dbbb870d2c" not found
Nov 03 00:05:06.586335 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:05:06.585950    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-provider-credentials podName:596732d7-7a22-4c49-81c7-e76f5694077a nodeName:}" failed. No retries permitted until 2026-11-03 00:05:08.58593234 +0000 UTC m=+4348.154287109 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-provider-credentials") pod "payload-processing-external-model-795b95959c-rpkc9" (UID: "596732d7-7a22-4c49-81c7-e76f5694077a") : secret "agc-cred-f2dbbb870d2c" not found
Nov 03 00:05:07.453478 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:07.453447    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:07.539093 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:07.539065    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh"]
Nov 03 00:05:07.904615 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:07.904577    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:08.076672 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:08.076627    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:08.610965 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:05:08.610925    4294 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f2dbbb870d2c: secret "agc-cred-f2dbbb870d2c" not found
Nov 03 00:05:08.610965 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:05:08.610948    4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-795b95959c-rpkc9: secret "agc-cred-f2dbbb870d2c" not found
Nov 03 00:05:08.611222 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:05:08.611010    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-provider-credentials podName:596732d7-7a22-4c49-81c7-e76f5694077a nodeName:}" failed. No retries permitted until 2026-11-03 00:05:12.610992811 +0000 UTC m=+4352.179347593 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-provider-credentials") pod "payload-processing-external-model-795b95959c-rpkc9" (UID: "596732d7-7a22-4c49-81c7-e76f5694077a") : secret "agc-cred-f2dbbb870d2c" not found
Nov 03 00:05:09.141584 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:09.141549    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:11.000474 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:11.000428    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9" podUID="596732d7-7a22-4c49-81c7-e76f5694077a" containerName="payload-processing" containerID="cri-o://34a87cae71216f90c8d771a1ab36a06e117dc25cc2b3ee027c2a6c000a7b6a99" gracePeriod=25
Nov 03 00:05:11.009677 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:11.009647    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:11.183445 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:11.183413    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:11.275386 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:11.275344    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"
Nov 03 00:05:11.341899 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:11.341837    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/596732d7-7a22-4c49-81c7-e76f5694077a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/596732d7-7a22-4c49-81c7-e76f5694077a-plugins-config-volume\") pod \"596732d7-7a22-4c49-81c7-e76f5694077a\" (UID: \"596732d7-7a22-4c49-81c7-e76f5694077a\") "
Nov 03 00:05:11.344031 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:11.343953    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596732d7-7a22-4c49-81c7-e76f5694077a-plugins-config-volume" pod "596732d7-7a22-4c49-81c7-e76f5694077a" (UID: "596732d7-7a22-4c49-81c7-e76f5694077a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:05:11.344414 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:11.344322    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/596732d7-7a22-4c49-81c7-e76f5694077a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/596732d7-7a22-4c49-81c7-e76f5694077a-routing-overlay-volume\") pod \"596732d7-7a22-4c49-81c7-e76f5694077a\" (UID: \"596732d7-7a22-4c49-81c7-e76f5694077a\") "
Nov 03 00:05:11.346571 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:11.345137    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596732d7-7a22-4c49-81c7-e76f5694077a-routing-overlay-volume" pod "596732d7-7a22-4c49-81c7-e76f5694077a" (UID: "596732d7-7a22-4c49-81c7-e76f5694077a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:05:11.346972 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:11.346897    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-provider-credentials\") pod \"596732d7-7a22-4c49-81c7-e76f5694077a\" (UID: \"596732d7-7a22-4c49-81c7-e76f5694077a\") "
Nov 03 00:05:11.347804 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:11.347744    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-kube-api-access-j647q\" (UniqueName: \"kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-kube-api-access-j647q\") pod \"596732d7-7a22-4c49-81c7-e76f5694077a\" (UID: \"596732d7-7a22-4c49-81c7-e76f5694077a\") "
Nov 03 00:05:11.349104 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:11.349055    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/596732d7-7a22-4c49-81c7-e76f5694077a-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:11.349104 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:11.349081    4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/596732d7-7a22-4c49-81c7-e76f5694077a-routing-overlay-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:11.353547 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:11.353521    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-kube-api-access-j647q" pod "596732d7-7a22-4c49-81c7-e76f5694077a" (UID: "596732d7-7a22-4c49-81c7-e76f5694077a"). InnerVolumeSpecName "kube-api-access-j647q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:05:11.353669 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:11.353633    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-provider-credentials" pod "596732d7-7a22-4c49-81c7-e76f5694077a" (UID: "596732d7-7a22-4c49-81c7-e76f5694077a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:05:11.450450 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:11.450413    4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-provider-credentials\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:11.450450 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:11.450439    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j647q\" (UniqueName: \"kubernetes.io/projected/596732d7-7a22-4c49-81c7-e76f5694077a-kube-api-access-j647q\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:11.805577 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:05:11.805537    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:05:12.040146 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:12.040111    4294 generic.go:356] "Generic (PLEG): container finished" podID="596732d7-7a22-4c49-81c7-e76f5694077a" containerID="34a87cae71216f90c8d771a1ab36a06e117dc25cc2b3ee027c2a6c000a7b6a99" exitCode=0
Nov 03 00:05:12.040575 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:12.040157    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"
Nov 03 00:05:12.040575 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:12.040164    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9" event={"ID":"596732d7-7a22-4c49-81c7-e76f5694077a","Type":"ContainerDied","Data":"34a87cae71216f90c8d771a1ab36a06e117dc25cc2b3ee027c2a6c000a7b6a99"}
Nov 03 00:05:12.040575 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:12.040293    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-795b95959c-rpkc9" event={"ID":"596732d7-7a22-4c49-81c7-e76f5694077a","Type":"ContainerDied","Data":"1e9bb07eae63a5b4a61741ceb6dcec984058e033ef98cc73ed431f1fe9243298"}
Nov 03 00:05:12.040575 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:12.040328    4294 scope.go:122] "RemoveContainer" containerID="34a87cae71216f90c8d771a1ab36a06e117dc25cc2b3ee027c2a6c000a7b6a99"
Nov 03 00:05:12.050940 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:12.050919    4294 scope.go:122] "RemoveContainer" containerID="34a87cae71216f90c8d771a1ab36a06e117dc25cc2b3ee027c2a6c000a7b6a99"
Nov 03 00:05:12.051266 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:05:12.051246    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34a87cae71216f90c8d771a1ab36a06e117dc25cc2b3ee027c2a6c000a7b6a99\": container with ID starting with 34a87cae71216f90c8d771a1ab36a06e117dc25cc2b3ee027c2a6c000a7b6a99 not found: ID does not exist" containerID="34a87cae71216f90c8d771a1ab36a06e117dc25cc2b3ee027c2a6c000a7b6a99"
Nov 03 00:05:12.051346 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:12.051274    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34a87cae71216f90c8d771a1ab36a06e117dc25cc2b3ee027c2a6c000a7b6a99"} err="failed to get container status \"34a87cae71216f90c8d771a1ab36a06e117dc25cc2b3ee027c2a6c000a7b6a99\": rpc error: code = NotFound desc = could not find container \"34a87cae71216f90c8d771a1ab36a06e117dc25cc2b3ee027c2a6c000a7b6a99\": container with ID starting with 34a87cae71216f90c8d771a1ab36a06e117dc25cc2b3ee027c2a6c000a7b6a99 not found: ID does not exist"
Nov 03 00:05:12.203892 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:12.203816    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"]
Nov 03 00:05:12.251700 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:12.251668    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-795b95959c-rpkc9"]
Nov 03 00:05:12.540408 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:12.540370    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh" podUID="c2d57a6a-c210-4895-8124-53a2514d1d3c" containerName="payload-processing" containerID="cri-o://44840256b8d07665a8d82be1157ad8f89b8385bf13e21a70a35d101c713332a8" gracePeriod=25
Nov 03 00:05:12.549930 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:12.549903    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="596732d7-7a22-4c49-81c7-e76f5694077a" path="/var/lib/kubelet/pods/596732d7-7a22-4c49-81c7-e76f5694077a/volumes"
Nov 03 00:05:12.814965 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:12.814934    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh"
Nov 03 00:05:12.978694 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:12.978656    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2d57a6a-c210-4895-8124-53a2514d1d3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d57a6a-c210-4895-8124-53a2514d1d3c-plugins-config-volume\") pod \"c2d57a6a-c210-4895-8124-53a2514d1d3c\" (UID: \"c2d57a6a-c210-4895-8124-53a2514d1d3c\") "
Nov 03 00:05:12.978907 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:12.978711    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d57a6a-c210-4895-8124-53a2514d1d3c-kube-api-access-rdcm4\" (UniqueName: \"kubernetes.io/projected/c2d57a6a-c210-4895-8124-53a2514d1d3c-kube-api-access-rdcm4\") pod \"c2d57a6a-c210-4895-8124-53a2514d1d3c\" (UID: \"c2d57a6a-c210-4895-8124-53a2514d1d3c\") "
Nov 03 00:05:12.979038 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:12.979015    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d57a6a-c210-4895-8124-53a2514d1d3c-plugins-config-volume" pod "c2d57a6a-c210-4895-8124-53a2514d1d3c" (UID: "c2d57a6a-c210-4895-8124-53a2514d1d3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:05:12.979445 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:12.979424    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d57a6a-c210-4895-8124-53a2514d1d3c-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:12.980854 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:12.980834    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d57a6a-c210-4895-8124-53a2514d1d3c-kube-api-access-rdcm4" pod "c2d57a6a-c210-4895-8124-53a2514d1d3c" (UID: "c2d57a6a-c210-4895-8124-53a2514d1d3c"). InnerVolumeSpecName "kube-api-access-rdcm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:05:13.053273 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.053187    4294 generic.go:356] "Generic (PLEG): container finished" podID="c2d57a6a-c210-4895-8124-53a2514d1d3c" containerID="44840256b8d07665a8d82be1157ad8f89b8385bf13e21a70a35d101c713332a8" exitCode=0
Nov 03 00:05:13.053273 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.053225    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh" event={"ID":"c2d57a6a-c210-4895-8124-53a2514d1d3c","Type":"ContainerDied","Data":"44840256b8d07665a8d82be1157ad8f89b8385bf13e21a70a35d101c713332a8"}
Nov 03 00:05:13.053273 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.053245    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh"
Nov 03 00:05:13.053273 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.053273    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh" event={"ID":"c2d57a6a-c210-4895-8124-53a2514d1d3c","Type":"ContainerDied","Data":"75eb5ea37aad40515677a64ac92d06a3cf9b74e5cd23a049d4bcea0b752ea740"}
Nov 03 00:05:13.053893 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.053296    4294 scope.go:122] "RemoveContainer" containerID="44840256b8d07665a8d82be1157ad8f89b8385bf13e21a70a35d101c713332a8"
Nov 03 00:05:13.064712 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.064456    4294 scope.go:122] "RemoveContainer" containerID="44840256b8d07665a8d82be1157ad8f89b8385bf13e21a70a35d101c713332a8"
Nov 03 00:05:13.064837 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:05:13.064808    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44840256b8d07665a8d82be1157ad8f89b8385bf13e21a70a35d101c713332a8\": container with ID starting with 44840256b8d07665a8d82be1157ad8f89b8385bf13e21a70a35d101c713332a8 not found: ID does not exist" containerID="44840256b8d07665a8d82be1157ad8f89b8385bf13e21a70a35d101c713332a8"
Nov 03 00:05:13.064895 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.064851    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44840256b8d07665a8d82be1157ad8f89b8385bf13e21a70a35d101c713332a8"} err="failed to get container status \"44840256b8d07665a8d82be1157ad8f89b8385bf13e21a70a35d101c713332a8\": rpc error: code = NotFound desc = could not find container \"44840256b8d07665a8d82be1157ad8f89b8385bf13e21a70a35d101c713332a8\": container with ID starting with 44840256b8d07665a8d82be1157ad8f89b8385bf13e21a70a35d101c713332a8 not found: ID does not exist"
Nov 03 00:05:13.080719 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.080686    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdcm4\" (UniqueName: \"kubernetes.io/projected/c2d57a6a-c210-4895-8124-53a2514d1d3c-kube-api-access-rdcm4\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:13.221631 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.221597    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh"]
Nov 03 00:05:13.224596 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.224566    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-666a38-6f5877bcb9-857kh"]
Nov 03 00:05:13.298492 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.298457    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:13.756408 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.756374    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:13.780372 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.780335    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-795b95959c-rszsl"]
Nov 03 00:05:13.781013 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.780998    4294 cpu_manager.go:414] "containerMap: removing container" podUID="596732d7-7a22-4c49-81c7-e76f5694077a" containerName="payload-processing"
Nov 03 00:05:13.781074 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.781014    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596732d7-7a22-4c49-81c7-e76f5694077a" containerName="payload-processing"
Nov 03 00:05:13.781074 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.781027    4294 cpu_manager.go:414] "containerMap: removing container" podUID="c2d57a6a-c210-4895-8124-53a2514d1d3c" containerName="payload-processing"
Nov 03 00:05:13.781074 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.781033    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d57a6a-c210-4895-8124-53a2514d1d3c" containerName="payload-processing"
Nov 03 00:05:13.781175 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.781152    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="596732d7-7a22-4c49-81c7-e76f5694077a" containerName="payload-processing"
Nov 03 00:05:13.781175 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.781160    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d57a6a-c210-4895-8124-53a2514d1d3c" containerName="payload-processing"
Nov 03 00:05:13.785927 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.785906    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl"
Nov 03 00:05:13.836086 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.836051    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f2dbbb870d2c\""
Nov 03 00:05:13.836086 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.836085    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 00:05:13.836329 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.836241    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 00:05:13.836329 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.836288    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 00:05:13.837338 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.837315    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-795b95959c-rszsl"]
Nov 03 00:05:13.882787 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.882298    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 00:05:13.882993 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.882940    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-phff6\""
Nov 03 00:05:13.889356 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.889324    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-routing-overlay-volume\") pod \"payload-processing-external-model-795b95959c-rszsl\" (UID: \"27de00d5-31cd-4f1c-9d58-24dba5e5afe4\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl"
Nov 03 00:05:13.889571 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.889373    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-plugins-config-volume\") pod \"payload-processing-external-model-795b95959c-rszsl\" (UID: \"27de00d5-31cd-4f1c-9d58-24dba5e5afe4\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl"
Nov 03 00:05:13.889639 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.889622    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-provider-credentials\") pod \"payload-processing-external-model-795b95959c-rszsl\" (UID: \"27de00d5-31cd-4f1c-9d58-24dba5e5afe4\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl"
Nov 03 00:05:13.889700 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.889678    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnppm\" (UniqueName: \"kubernetes.io/projected/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-kube-api-access-bnppm\") pod \"payload-processing-external-model-795b95959c-rszsl\" (UID: \"27de00d5-31cd-4f1c-9d58-24dba5e5afe4\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl"
Nov 03 00:05:13.990880 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.990838    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-provider-credentials\") pod \"payload-processing-external-model-795b95959c-rszsl\" (UID: \"27de00d5-31cd-4f1c-9d58-24dba5e5afe4\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl"
Nov 03 00:05:13.991100 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.990887    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnppm\" (UniqueName: \"kubernetes.io/projected/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-kube-api-access-bnppm\") pod \"payload-processing-external-model-795b95959c-rszsl\" (UID: \"27de00d5-31cd-4f1c-9d58-24dba5e5afe4\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl"
Nov 03 00:05:13.991100 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.991083    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-routing-overlay-volume\") pod \"payload-processing-external-model-795b95959c-rszsl\" (UID: \"27de00d5-31cd-4f1c-9d58-24dba5e5afe4\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl"
Nov 03 00:05:13.991188 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.991119    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-plugins-config-volume\") pod \"payload-processing-external-model-795b95959c-rszsl\" (UID: \"27de00d5-31cd-4f1c-9d58-24dba5e5afe4\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl"
Nov 03 00:05:13.991771 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.991748    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-routing-overlay-volume\") pod \"payload-processing-external-model-795b95959c-rszsl\" (UID: \"27de00d5-31cd-4f1c-9d58-24dba5e5afe4\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl"
Nov 03 00:05:13.991908 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.991885    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-plugins-config-volume\") pod \"payload-processing-external-model-795b95959c-rszsl\" (UID: \"27de00d5-31cd-4f1c-9d58-24dba5e5afe4\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl"
Nov 03 00:05:13.993374 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:13.993353    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-provider-credentials\") pod \"payload-processing-external-model-795b95959c-rszsl\" (UID: \"27de00d5-31cd-4f1c-9d58-24dba5e5afe4\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl"
Nov 03 00:05:14.045568 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:14.045487    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnppm\" (UniqueName: \"kubernetes.io/projected/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-kube-api-access-bnppm\") pod \"payload-processing-external-model-795b95959c-rszsl\" (UID: \"27de00d5-31cd-4f1c-9d58-24dba5e5afe4\") " pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl"
Nov 03 00:05:14.100379 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:14.100338    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl"
Nov 03 00:05:14.500279 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:14.500245    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-795b95959c-rszsl"]
Nov 03 00:05:14.500417 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:05:14.500399    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27de00d5_31cd_4f1c_9d58_24dba5e5afe4.slice/crio-9e37884575d664a88d015ac8e05467c7e8bdcbdfa4e192c4b28200ebd7810bca WatchSource:0}: Error finding container 9e37884575d664a88d015ac8e05467c7e8bdcbdfa4e192c4b28200ebd7810bca: Status 404 returned error can't find the container with id 9e37884575d664a88d015ac8e05467c7e8bdcbdfa4e192c4b28200ebd7810bca
Nov 03 00:05:14.550134 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:14.550106    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d57a6a-c210-4895-8124-53a2514d1d3c" path="/var/lib/kubelet/pods/c2d57a6a-c210-4895-8124-53a2514d1d3c/volumes"
Nov 03 00:05:15.068674 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:15.068638    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl" event={"ID":"27de00d5-31cd-4f1c-9d58-24dba5e5afe4","Type":"ContainerStarted","Data":"c62217033502e6888ef90227a6410a5102ac951974345e571062fed02420108f"}
Nov 03 00:05:15.068893 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:15.068698    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl" event={"ID":"27de00d5-31cd-4f1c-9d58-24dba5e5afe4","Type":"ContainerStarted","Data":"9e37884575d664a88d015ac8e05467c7e8bdcbdfa4e192c4b28200ebd7810bca"}
Nov 03 00:05:15.068893 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:15.068720    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl"
Nov 03 00:05:15.180833 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:15.180776    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl" podStartSLOduration=2.180760218 podStartE2EDuration="2.180760218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:15.179075274 +0000 UTC m=+4354.747430062" watchObservedRunningTime="2026-11-03 00:05:15.180760218 +0000 UTC m=+4354.749115005"
Nov 03 00:05:15.916292 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:15.916129    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:16.090482 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:16.090425    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:17.366413 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:17.366376    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:17.452616 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:17.452571    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-795b95959c-rszsl"]
Nov 03 00:05:17.529871 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:17.529836    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:17.691923 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:17.691837    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:17.864171 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:17.864141    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:18.521973 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:18.521942    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:18.674691 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:18.674661    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:19.263288 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:19.263248    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:19.457508 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:19.457477    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:22.453817 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:22.453774    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl" podUID="27de00d5-31cd-4f1c-9d58-24dba5e5afe4" containerName="payload-processing" containerID="cri-o://c62217033502e6888ef90227a6410a5102ac951974345e571062fed02420108f" gracePeriod=25
Nov 03 00:05:22.726358 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:22.726332    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl"
Nov 03 00:05:22.835574 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:22.835533    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-provider-credentials\") pod \"27de00d5-31cd-4f1c-9d58-24dba5e5afe4\" (UID: \"27de00d5-31cd-4f1c-9d58-24dba5e5afe4\") "
Nov 03 00:05:22.835748 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:22.835611    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-kube-api-access-bnppm\" (UniqueName: \"kubernetes.io/projected/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-kube-api-access-bnppm\") pod \"27de00d5-31cd-4f1c-9d58-24dba5e5afe4\" (UID: \"27de00d5-31cd-4f1c-9d58-24dba5e5afe4\") "
Nov 03 00:05:22.835748 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:22.835639    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-routing-overlay-volume\") pod \"27de00d5-31cd-4f1c-9d58-24dba5e5afe4\" (UID: \"27de00d5-31cd-4f1c-9d58-24dba5e5afe4\") "
Nov 03 00:05:22.835748 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:22.835698    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-plugins-config-volume\") pod \"27de00d5-31cd-4f1c-9d58-24dba5e5afe4\" (UID: \"27de00d5-31cd-4f1c-9d58-24dba5e5afe4\") "
Nov 03 00:05:22.836166 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:22.836141    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-routing-overlay-volume" pod "27de00d5-31cd-4f1c-9d58-24dba5e5afe4" (UID: "27de00d5-31cd-4f1c-9d58-24dba5e5afe4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:05:22.836407 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:22.836212    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-plugins-config-volume" pod "27de00d5-31cd-4f1c-9d58-24dba5e5afe4" (UID: "27de00d5-31cd-4f1c-9d58-24dba5e5afe4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:05:22.838596 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:22.838188    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-provider-credentials" pod "27de00d5-31cd-4f1c-9d58-24dba5e5afe4" (UID: "27de00d5-31cd-4f1c-9d58-24dba5e5afe4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:05:22.838596 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:22.838289    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-kube-api-access-bnppm" pod "27de00d5-31cd-4f1c-9d58-24dba5e5afe4" (UID: "27de00d5-31cd-4f1c-9d58-24dba5e5afe4"). InnerVolumeSpecName "kube-api-access-bnppm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:05:22.937649 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:22.937613    4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-provider-credentials\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:22.937649 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:22.937644    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnppm\" (UniqueName: \"kubernetes.io/projected/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-kube-api-access-bnppm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:22.937649 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:22.937653    4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-routing-overlay-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:22.937904 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:22.937662    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27de00d5-31cd-4f1c-9d58-24dba5e5afe4-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:23.129233 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:23.129202    4294 generic.go:356] "Generic (PLEG): container finished" podID="27de00d5-31cd-4f1c-9d58-24dba5e5afe4" containerID="c62217033502e6888ef90227a6410a5102ac951974345e571062fed02420108f" exitCode=0
Nov 03 00:05:23.129427 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:23.129256    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl"
Nov 03 00:05:23.129427 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:23.129263    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl" event={"ID":"27de00d5-31cd-4f1c-9d58-24dba5e5afe4","Type":"ContainerDied","Data":"c62217033502e6888ef90227a6410a5102ac951974345e571062fed02420108f"}
Nov 03 00:05:23.129427 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:23.129382    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-795b95959c-rszsl" event={"ID":"27de00d5-31cd-4f1c-9d58-24dba5e5afe4","Type":"ContainerDied","Data":"9e37884575d664a88d015ac8e05467c7e8bdcbdfa4e192c4b28200ebd7810bca"}
Nov 03 00:05:23.129427 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:23.129411    4294 scope.go:122] "RemoveContainer" containerID="c62217033502e6888ef90227a6410a5102ac951974345e571062fed02420108f"
Nov 03 00:05:23.140339 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:23.140320    4294 scope.go:122] "RemoveContainer" containerID="c62217033502e6888ef90227a6410a5102ac951974345e571062fed02420108f"
Nov 03 00:05:23.140655 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:05:23.140634    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c62217033502e6888ef90227a6410a5102ac951974345e571062fed02420108f\": container with ID starting with c62217033502e6888ef90227a6410a5102ac951974345e571062fed02420108f not found: ID does not exist" containerID="c62217033502e6888ef90227a6410a5102ac951974345e571062fed02420108f"
Nov 03 00:05:23.140740 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:23.140664    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c62217033502e6888ef90227a6410a5102ac951974345e571062fed02420108f"} err="failed to get container status \"c62217033502e6888ef90227a6410a5102ac951974345e571062fed02420108f\": rpc error: code = NotFound desc = could not find container \"c62217033502e6888ef90227a6410a5102ac951974345e571062fed02420108f\": container with ID starting with c62217033502e6888ef90227a6410a5102ac951974345e571062fed02420108f not found: ID does not exist"
Nov 03 00:05:23.286857 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:23.286826    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-795b95959c-rszsl"]
Nov 03 00:05:23.292146 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:23.292117    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-795b95959c-rszsl"]
Nov 03 00:05:23.296052 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:23.296021    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:24.472704 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:24.472622    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq"]
Nov 03 00:05:24.548920 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:24.548891    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27de00d5-31cd-4f1c-9d58-24dba5e5afe4" path="/var/lib/kubelet/pods/27de00d5-31cd-4f1c-9d58-24dba5e5afe4/volumes"
Nov 03 00:05:24.589160 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:24.589127    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:24.756599 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:24.756286    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:24.921301 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:24.921267    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:26.187002 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:26.186249    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:28.480069 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:28.480037    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:29.473714 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:29.473664    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq" podUID="4b964a56-7099-408e-97dd-71a707fdacbd" containerName="payload-pre-processing" containerID="cri-o://acdf3817fbf863fe43db5b952d067797348e3b35eb5fa22100c622f33010e7d9" gracePeriod=25
Nov 03 00:05:29.737846 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:29.737788    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq"
Nov 03 00:05:29.910534 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:29.910504    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b964a56-7099-408e-97dd-71a707fdacbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b964a56-7099-408e-97dd-71a707fdacbd-plugins-config-volume\") pod \"4b964a56-7099-408e-97dd-71a707fdacbd\" (UID: \"4b964a56-7099-408e-97dd-71a707fdacbd\") "
Nov 03 00:05:29.910720 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:29.910706    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b964a56-7099-408e-97dd-71a707fdacbd-kube-api-access-cmp7x\" (UniqueName: \"kubernetes.io/projected/4b964a56-7099-408e-97dd-71a707fdacbd-kube-api-access-cmp7x\") pod \"4b964a56-7099-408e-97dd-71a707fdacbd\" (UID: \"4b964a56-7099-408e-97dd-71a707fdacbd\") "
Nov 03 00:05:29.910937 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:29.910913    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b964a56-7099-408e-97dd-71a707fdacbd-plugins-config-volume" pod "4b964a56-7099-408e-97dd-71a707fdacbd" (UID: "4b964a56-7099-408e-97dd-71a707fdacbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:05:29.911301 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:29.911282    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b964a56-7099-408e-97dd-71a707fdacbd-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:29.912690 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:29.912672    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b964a56-7099-408e-97dd-71a707fdacbd-kube-api-access-cmp7x" pod "4b964a56-7099-408e-97dd-71a707fdacbd" (UID: "4b964a56-7099-408e-97dd-71a707fdacbd"). InnerVolumeSpecName "kube-api-access-cmp7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:05:30.012490 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:30.012454    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmp7x\" (UniqueName: \"kubernetes.io/projected/4b964a56-7099-408e-97dd-71a707fdacbd-kube-api-access-cmp7x\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:30.145999 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:30.145959    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:30.224483 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:30.224452    4294 generic.go:356] "Generic (PLEG): container finished" podID="4b964a56-7099-408e-97dd-71a707fdacbd" containerID="acdf3817fbf863fe43db5b952d067797348e3b35eb5fa22100c622f33010e7d9" exitCode=0
Nov 03 00:05:30.224663 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:30.224499    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq"
Nov 03 00:05:30.224663 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:30.224509    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq" event={"ID":"4b964a56-7099-408e-97dd-71a707fdacbd","Type":"ContainerDied","Data":"acdf3817fbf863fe43db5b952d067797348e3b35eb5fa22100c622f33010e7d9"}
Nov 03 00:05:30.224663 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:30.224549    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq" event={"ID":"4b964a56-7099-408e-97dd-71a707fdacbd","Type":"ContainerDied","Data":"2d88500f1e4b7a317b2a36553c07e3b38139ad2d8937475cb44c75c4656bd654"}
Nov 03 00:05:30.224663 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:30.224570    4294 scope.go:122] "RemoveContainer" containerID="acdf3817fbf863fe43db5b952d067797348e3b35eb5fa22100c622f33010e7d9"
Nov 03 00:05:30.234509 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:30.234489    4294 scope.go:122] "RemoveContainer" containerID="acdf3817fbf863fe43db5b952d067797348e3b35eb5fa22100c622f33010e7d9"
Nov 03 00:05:30.234767 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:05:30.234747    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acdf3817fbf863fe43db5b952d067797348e3b35eb5fa22100c622f33010e7d9\": container with ID starting with acdf3817fbf863fe43db5b952d067797348e3b35eb5fa22100c622f33010e7d9 not found: ID does not exist" containerID="acdf3817fbf863fe43db5b952d067797348e3b35eb5fa22100c622f33010e7d9"
Nov 03 00:05:30.234807 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:30.234775    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acdf3817fbf863fe43db5b952d067797348e3b35eb5fa22100c622f33010e7d9"} err="failed to get container status \"acdf3817fbf863fe43db5b952d067797348e3b35eb5fa22100c622f33010e7d9\": rpc error: code = NotFound desc = could not find container \"acdf3817fbf863fe43db5b952d067797348e3b35eb5fa22100c622f33010e7d9\": container with ID starting with acdf3817fbf863fe43db5b952d067797348e3b35eb5fa22100c622f33010e7d9 not found: ID does not exist"
Nov 03 00:05:30.381611 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:30.381534    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq"]
Nov 03 00:05:30.428922 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:30.428891    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9d4fc-9b76796ff-65rnq"]
Nov 03 00:05:30.548399 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:30.548359    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b964a56-7099-408e-97dd-71a707fdacbd" path="/var/lib/kubelet/pods/4b964a56-7099-408e-97dd-71a707fdacbd/volumes"
Nov 03 00:05:32.137457 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:32.137426    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:32.308074 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:32.308033    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2"]
Nov 03 00:05:32.589621 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:32.589593    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:32.771554 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:32.771513    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:33.495483 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.495449    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:33.841370 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.841280    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"]
Nov 03 00:05:33.842058 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.842037    4294 cpu_manager.go:414] "containerMap: removing container" podUID="27de00d5-31cd-4f1c-9d58-24dba5e5afe4" containerName="payload-processing"
Nov 03 00:05:33.842058 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.842055    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27de00d5-31cd-4f1c-9d58-24dba5e5afe4" containerName="payload-processing"
Nov 03 00:05:33.842250 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.842082    4294 cpu_manager.go:414] "containerMap: removing container" podUID="4b964a56-7099-408e-97dd-71a707fdacbd" containerName="payload-pre-processing"
Nov 03 00:05:33.842250 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.842087    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b964a56-7099-408e-97dd-71a707fdacbd" containerName="payload-pre-processing"
Nov 03 00:05:33.842250 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.842196    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b964a56-7099-408e-97dd-71a707fdacbd" containerName="payload-pre-processing"
Nov 03 00:05:33.842250 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.842211    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="27de00d5-31cd-4f1c-9d58-24dba5e5afe4" containerName="payload-processing"
Nov 03 00:05:33.847044 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.847023    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:33.879852 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.879810    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:33.880059 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.879872    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f17fbbad-f397-418e-a74c-0ce90afd568b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:33.880226 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.880202    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:33.880345 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.880326    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:33.880387 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.880360    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:33.880430 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.880388    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9fft\" (UniqueName: \"kubernetes.io/projected/f17fbbad-f397-418e-a74c-0ce90afd568b-kube-api-access-f9fft\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:33.880430 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.880410    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:33.897214 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.897184    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 00:05:33.944749 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.944721    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 00:05:33.944902 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.944726    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tbzbd\""
Nov 03 00:05:33.944902 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.944724    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 00:05:33.949046 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.949021    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"]
Nov 03 00:05:33.981937 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.981904    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:33.982120 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.981952    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f17fbbad-f397-418e-a74c-0ce90afd568b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:33.982120 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.982102    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:33.982218 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.982197    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:33.982268 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.982232    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:33.982327 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.982272    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9fft\" (UniqueName: \"kubernetes.io/projected/f17fbbad-f397-418e-a74c-0ce90afd568b-kube-api-access-f9fft\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:33.982327 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.982305    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:33.982913 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.982883    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:33.984358 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.984340    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f17fbbad-f397-418e-a74c-0ce90afd568b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:33.984458 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.984408    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:33.984807 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.984788    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:33.984877 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:33.984806    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:34.036628 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:34.036595    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:34.036821 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:34.036798    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9fft\" (UniqueName: \"kubernetes.io/projected/f17fbbad-f397-418e-a74c-0ce90afd568b-kube-api-access-f9fft\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rbl47\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:34.162271 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:34.162188    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:34.584286 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:05:34.584247    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf17fbbad_f397_418e_a74c_0ce90afd568b.slice/crio-4fc3e748113639428da115e4e5002ca294bda778e44c113e484a85f7f529a593 WatchSource:0}: Error finding container 4fc3e748113639428da115e4e5002ca294bda778e44c113e484a85f7f529a593: Status 404 returned error can't find the container with id 4fc3e748113639428da115e4e5002ca294bda778e44c113e484a85f7f529a593
Nov 03 00:05:34.584725 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:34.584384    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"]
Nov 03 00:05:34.587019 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:34.586949    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:05:34.587099 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:34.587029    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:05:35.258446 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:35.258413    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47" event={"ID":"f17fbbad-f397-418e-a74c-0ce90afd568b","Type":"ContainerStarted","Data":"35b5e0f88323304565f30863ae220fd4f81da86007fed5236d572bcc93658894"}
Nov 03 00:05:35.258596 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:35.258459    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47" event={"ID":"f17fbbad-f397-418e-a74c-0ce90afd568b","Type":"ContainerStarted","Data":"4fc3e748113639428da115e4e5002ca294bda778e44c113e484a85f7f529a593"}
Nov 03 00:05:35.258596 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:35.258584    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:35.260545 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:35.260519    4294 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rbl47 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:05:35.260696 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:35.260583    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47" podUID="f17fbbad-f397-418e-a74c-0ce90afd568b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:05:35.457629 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:35.457522    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47" podStartSLOduration=2.457507504 podStartE2EDuration="2.457507504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:35.416900631 +0000 UTC m=+4374.985255416" watchObservedRunningTime="2026-11-03 00:05:35.457507504 +0000 UTC m=+4375.025862288"
Nov 03 00:05:35.670933 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:35.670898    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:36.276436 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:36.276399    4294 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rbl47 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:05:36.276697 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:36.276672    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47" podUID="f17fbbad-f397-418e-a74c-0ce90afd568b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:05:37.308528 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:37.308478    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2" podUID="75f869b3-35bd-4168-bdd5-8a6334bc8e95" containerName="payload-pre-processing" containerID="cri-o://4165c188da3d3d0c72c1f2bd68399b92acc653a952b679fa612593db5b3f8e47" gracePeriod=25
Nov 03 00:05:37.564766 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:37.564703    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2"
Nov 03 00:05:37.627721 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:37.627692    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f869b3-35bd-4168-bdd5-8a6334bc8e95-kube-api-access-5xx7p\" (UniqueName: \"kubernetes.io/projected/75f869b3-35bd-4168-bdd5-8a6334bc8e95-kube-api-access-5xx7p\") pod \"75f869b3-35bd-4168-bdd5-8a6334bc8e95\" (UID: \"75f869b3-35bd-4168-bdd5-8a6334bc8e95\") "
Nov 03 00:05:37.627907 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:37.627777    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f869b3-35bd-4168-bdd5-8a6334bc8e95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75f869b3-35bd-4168-bdd5-8a6334bc8e95-plugins-config-volume\") pod \"75f869b3-35bd-4168-bdd5-8a6334bc8e95\" (UID: \"75f869b3-35bd-4168-bdd5-8a6334bc8e95\") "
Nov 03 00:05:37.628220 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:37.628196    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f869b3-35bd-4168-bdd5-8a6334bc8e95-plugins-config-volume" pod "75f869b3-35bd-4168-bdd5-8a6334bc8e95" (UID: "75f869b3-35bd-4168-bdd5-8a6334bc8e95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:05:37.628466 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:37.628445    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75f869b3-35bd-4168-bdd5-8a6334bc8e95-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:37.629916 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:37.629894    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f869b3-35bd-4168-bdd5-8a6334bc8e95-kube-api-access-5xx7p" pod "75f869b3-35bd-4168-bdd5-8a6334bc8e95" (UID: "75f869b3-35bd-4168-bdd5-8a6334bc8e95"). InnerVolumeSpecName "kube-api-access-5xx7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:05:37.729806 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:37.729770    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xx7p\" (UniqueName: \"kubernetes.io/projected/75f869b3-35bd-4168-bdd5-8a6334bc8e95-kube-api-access-5xx7p\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:05:37.820639 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:37.820526    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:38.281207 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:38.281174    4294 generic.go:356] "Generic (PLEG): container finished" podID="75f869b3-35bd-4168-bdd5-8a6334bc8e95" containerID="4165c188da3d3d0c72c1f2bd68399b92acc653a952b679fa612593db5b3f8e47" exitCode=0
Nov 03 00:05:38.281390 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:38.281220    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2"
Nov 03 00:05:38.281390 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:38.281258    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2" event={"ID":"75f869b3-35bd-4168-bdd5-8a6334bc8e95","Type":"ContainerDied","Data":"4165c188da3d3d0c72c1f2bd68399b92acc653a952b679fa612593db5b3f8e47"}
Nov 03 00:05:38.281390 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:38.281307    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2" event={"ID":"75f869b3-35bd-4168-bdd5-8a6334bc8e95","Type":"ContainerDied","Data":"ca6b58aa5a9b06399420f9e7415503938376d8955c6324c480e644ae7010fd08"}
Nov 03 00:05:38.281390 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:38.281327    4294 scope.go:122] "RemoveContainer" containerID="4165c188da3d3d0c72c1f2bd68399b92acc653a952b679fa612593db5b3f8e47"
Nov 03 00:05:38.291740 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:38.291725    4294 scope.go:122] "RemoveContainer" containerID="4165c188da3d3d0c72c1f2bd68399b92acc653a952b679fa612593db5b3f8e47"
Nov 03 00:05:38.292021 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:05:38.291999    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4165c188da3d3d0c72c1f2bd68399b92acc653a952b679fa612593db5b3f8e47\": container with ID starting with 4165c188da3d3d0c72c1f2bd68399b92acc653a952b679fa612593db5b3f8e47 not found: ID does not exist" containerID="4165c188da3d3d0c72c1f2bd68399b92acc653a952b679fa612593db5b3f8e47"
Nov 03 00:05:38.292082 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:38.292033    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4165c188da3d3d0c72c1f2bd68399b92acc653a952b679fa612593db5b3f8e47"} err="failed to get container status \"4165c188da3d3d0c72c1f2bd68399b92acc653a952b679fa612593db5b3f8e47\": rpc error: code = NotFound desc = could not find container \"4165c188da3d3d0c72c1f2bd68399b92acc653a952b679fa612593db5b3f8e47\": container with ID starting with 4165c188da3d3d0c72c1f2bd68399b92acc653a952b679fa612593db5b3f8e47 not found: ID does not exist"
Nov 03 00:05:38.438764 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:38.438734    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2"]
Nov 03 00:05:38.441818 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:38.441796    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-68d89f-5fb5bc49f5-d9dr2"]
Nov 03 00:05:38.489259 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:38.489230    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:38.551055 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:38.550955    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f869b3-35bd-4168-bdd5-8a6334bc8e95" path="/var/lib/kubelet/pods/75f869b3-35bd-4168-bdd5-8a6334bc8e95/volumes"
Nov 03 00:05:39.269615 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:39.269586    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:05:41.476313 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:41.476250    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:41.754097 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:05:41.754060    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:05:43.244426 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:43.244395    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:46.285425 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:46.285391    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:47.750738 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:47.750660    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:52.515571 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:52.515535    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:57.048110 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:05:57.048077    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:00.021137 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:00.021083    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:06:00.079126 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:00.079071    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 03 00:06:00.085093 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:00.085050    4294 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j" podUID="f483c007-731c-42e7-a7f2-9b7877fbba41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:06:00.415410 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:00.415302    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j" podUID="f483c007-731c-42e7-a7f2-9b7877fbba41" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 00:06:01.857931 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:01.857870    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:06.434260 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:06.434228    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:11.028833 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:11.028796    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:11.750884 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:06:11.750844    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:06:15.544105 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:15.544073    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:20.147846 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:20.147813    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:24.641757 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:24.641727    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:29.425185 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:29.425152    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:32.035770 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:32.035734    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:33.467889 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:33.467855    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:34.457554 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:34.457517    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"]
Nov 03 00:06:34.457751 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:34.457722    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2" podUID="a6c764fb-09a2-4b5e-8588-b463f9487461" containerName="maas-api" containerID="cri-o://a31ba681616b0337db5344c4c4b6c0766ac5fa6078e1d65646b432e723c86838" gracePeriod=30
Nov 03 00:06:34.581540 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:34.581500    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:34.710511 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:34.710425    4294 generic.go:356] "Generic (PLEG): container finished" podID="a6c764fb-09a2-4b5e-8588-b463f9487461" containerID="a31ba681616b0337db5344c4c4b6c0766ac5fa6078e1d65646b432e723c86838" exitCode=0
Nov 03 00:06:34.710794 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:34.710526    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2" event={"ID":"a6c764fb-09a2-4b5e-8588-b463f9487461","Type":"ContainerDied","Data":"a31ba681616b0337db5344c4c4b6c0766ac5fa6078e1d65646b432e723c86838"}
Nov 03 00:06:34.742859 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:34.742831    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"
Nov 03 00:06:34.860075 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:34.860040    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6c764fb-09a2-4b5e-8588-b463f9487461-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6c764fb-09a2-4b5e-8588-b463f9487461-maas-api-tls\") pod \"a6c764fb-09a2-4b5e-8588-b463f9487461\" (UID: \"a6c764fb-09a2-4b5e-8588-b463f9487461\") "
Nov 03 00:06:34.860357 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:34.860341    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6c764fb-09a2-4b5e-8588-b463f9487461-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c764fb-09a2-4b5e-8588-b463f9487461-metrics-certs\") pod \"a6c764fb-09a2-4b5e-8588-b463f9487461\" (UID: \"a6c764fb-09a2-4b5e-8588-b463f9487461\") "
Nov 03 00:06:34.860819 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:34.860791    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6c764fb-09a2-4b5e-8588-b463f9487461-kube-api-access-s568v\" (UniqueName: \"kubernetes.io/projected/a6c764fb-09a2-4b5e-8588-b463f9487461-kube-api-access-s568v\") pod \"a6c764fb-09a2-4b5e-8588-b463f9487461\" (UID: \"a6c764fb-09a2-4b5e-8588-b463f9487461\") "
Nov 03 00:06:34.860922 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:34.860859    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6c764fb-09a2-4b5e-8588-b463f9487461-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c764fb-09a2-4b5e-8588-b463f9487461-openshift-service-ca\") pod \"a6c764fb-09a2-4b5e-8588-b463f9487461\" (UID: \"a6c764fb-09a2-4b5e-8588-b463f9487461\") "
Nov 03 00:06:34.861594 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:34.861387    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c764fb-09a2-4b5e-8588-b463f9487461-openshift-service-ca" pod "a6c764fb-09a2-4b5e-8588-b463f9487461" (UID: "a6c764fb-09a2-4b5e-8588-b463f9487461"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:06:34.861594 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:34.861569    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c764fb-09a2-4b5e-8588-b463f9487461-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:06:34.863760 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:34.863736    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c764fb-09a2-4b5e-8588-b463f9487461-kube-api-access-s568v" pod "a6c764fb-09a2-4b5e-8588-b463f9487461" (UID: "a6c764fb-09a2-4b5e-8588-b463f9487461"). InnerVolumeSpecName "kube-api-access-s568v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:06:34.871441 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:34.871409    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6c764fb-09a2-4b5e-8588-b463f9487461-metrics-certs" pod "a6c764fb-09a2-4b5e-8588-b463f9487461" (UID: "a6c764fb-09a2-4b5e-8588-b463f9487461"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:34.871441 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:34.871423    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6c764fb-09a2-4b5e-8588-b463f9487461-maas-api-tls" pod "a6c764fb-09a2-4b5e-8588-b463f9487461" (UID: "a6c764fb-09a2-4b5e-8588-b463f9487461"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:34.963353 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:34.963250    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6c764fb-09a2-4b5e-8588-b463f9487461-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:06:34.963353 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:34.963291    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c764fb-09a2-4b5e-8588-b463f9487461-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:06:34.963353 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:34.963310    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s568v\" (UniqueName: \"kubernetes.io/projected/a6c764fb-09a2-4b5e-8588-b463f9487461-kube-api-access-s568v\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:06:35.682812 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:35.682776    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:35.718272 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:35.718248    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"
Nov 03 00:06:35.718430 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:35.718287    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2" event={"ID":"a6c764fb-09a2-4b5e-8588-b463f9487461","Type":"ContainerDied","Data":"802572823f6f76256f3911792730b3e71e08a1a2cf7e3266ccd7df4e28700e7b"}
Nov 03 00:06:35.718430 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:35.718355    4294 scope.go:122] "RemoveContainer" containerID="a31ba681616b0337db5344c4c4b6c0766ac5fa6078e1d65646b432e723c86838"
Nov 03 00:06:35.855911 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:35.855855    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:35.880209 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:35.880177    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"]
Nov 03 00:06:35.925729 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:35.925704    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c9d4fc-bd8bcd94c-2ckj2"]
Nov 03 00:06:36.288474 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:36.288439    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:36.497498 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:36.497466    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:36.548410 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:36.548330    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6c764fb-09a2-4b5e-8588-b463f9487461" path="/var/lib/kubelet/pods/a6c764fb-09a2-4b5e-8588-b463f9487461/volumes"
Nov 03 00:06:36.669153 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:36.669123    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:37.175678 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:37.175627    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:37.654083 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:37.654049    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:38.368826 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:38.368787    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:38.611800 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:38.611768    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:39.050489 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:39.050459    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:40.009771 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:40.009739    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:40.175231 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:40.175202    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:41.346308 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:41.346277    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:41.752878 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:06:41.752841    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:06:44.177110 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:44.177079    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:44.326058 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:44.326022    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:46.137822 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:46.137789    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:46.335480 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:46.335448    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:46.515882 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:46.515848    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:46.701801 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:46.701768    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:49.767236 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:49.767197    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:51.053632 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:51.053601    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:55.631017 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:55.630910    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:59.129038 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:06:59.128999    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:00.253004 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:00.249690    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:04.876344 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:04.876310    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:05.196485 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:05.196395    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:05.643895 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:05.643862    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:05.822028 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:05.821967    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:06.011957 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:06.011916    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:09.545726 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:09.545693    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:11.751289 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:07:11.751247    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:07:14.048678 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:14.048643    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:16.646418 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:16.646385    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:18.967498 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:18.967464    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:23.539696 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:23.539659    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:28.045828 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:28.045791    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:32.609274 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:32.609236    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:37.144228 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:37.144189    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:41.111750 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:41.111720    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:41.754798 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:07:41.754757    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:07:42.521354 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:42.521324    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:42.735692 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:42.735654    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:42.936577 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:42.936500    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:43.177503 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:43.177472    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"]
Nov 03 00:07:43.177692 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:43.177677    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt" podUID="aff7037f-31de-4b57-a263-0229aa21b689" containerName="istio-proxy" containerID="cri-o://9a49544380cba41eaa1a865959bcf5e224d97928daa9ac92df8d8bd1cf719ec0" gracePeriod=30
Nov 03 00:07:43.218266 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:43.218190    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:43.517458 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:43.517426    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:43.715572 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:43.715540    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:44.653803 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:44.653760    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:44.927107 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:44.927024    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:46.958789 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:46.958753    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:47.971594 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:47.971558    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:48.402663 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.402633    4294 generic.go:356] "Generic (PLEG): container finished" podID="aff7037f-31de-4b57-a263-0229aa21b689" containerID="9a49544380cba41eaa1a865959bcf5e224d97928daa9ac92df8d8bd1cf719ec0" exitCode=0
Nov 03 00:07:48.402856 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.402711    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt" event={"ID":"aff7037f-31de-4b57-a263-0229aa21b689","Type":"ContainerDied","Data":"9a49544380cba41eaa1a865959bcf5e224d97928daa9ac92df8d8bd1cf719ec0"}
Nov 03 00:07:48.436007 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.435969    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:07:48.550104 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.550026    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff7037f-31de-4b57-a263-0229aa21b689-istio-token\" (UniqueName: \"kubernetes.io/projected/aff7037f-31de-4b57-a263-0229aa21b689-istio-token\") pod \"aff7037f-31de-4b57-a263-0229aa21b689\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") "
Nov 03 00:07:48.550277 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.550129    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-istio-data\") pod \"aff7037f-31de-4b57-a263-0229aa21b689\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") "
Nov 03 00:07:48.550277 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.550191    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-workload-socket\") pod \"aff7037f-31de-4b57-a263-0229aa21b689\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") "
Nov 03 00:07:48.550277 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.550252    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-credential-socket\") pod \"aff7037f-31de-4b57-a263-0229aa21b689\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") "
Nov 03 00:07:48.550471 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.550307    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff7037f-31de-4b57-a263-0229aa21b689-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff7037f-31de-4b57-a263-0229aa21b689-istiod-ca-cert\") pod \"aff7037f-31de-4b57-a263-0229aa21b689\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") "
Nov 03 00:07:48.550471 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.550368    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-workload-certs\") pod \"aff7037f-31de-4b57-a263-0229aa21b689\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") "
Nov 03 00:07:48.550471 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.550453    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-istio-data" pod "aff7037f-31de-4b57-a263-0229aa21b689" (UID: "aff7037f-31de-4b57-a263-0229aa21b689"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:07:48.550643 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.550463    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-istio-envoy\") pod \"aff7037f-31de-4b57-a263-0229aa21b689\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") "
Nov 03 00:07:48.550643 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.550534    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aff7037f-31de-4b57-a263-0229aa21b689-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff7037f-31de-4b57-a263-0229aa21b689-istio-podinfo\") pod \"aff7037f-31de-4b57-a263-0229aa21b689\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") "
Nov 03 00:07:48.550643 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.550632    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff7037f-31de-4b57-a263-0229aa21b689-kube-api-access-pplxx\" (UniqueName: \"kubernetes.io/projected/aff7037f-31de-4b57-a263-0229aa21b689-kube-api-access-pplxx\") pod \"aff7037f-31de-4b57-a263-0229aa21b689\" (UID: \"aff7037f-31de-4b57-a263-0229aa21b689\") "
Nov 03 00:07:48.550818 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.550655    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-workload-socket" pod "aff7037f-31de-4b57-a263-0229aa21b689" (UID: "aff7037f-31de-4b57-a263-0229aa21b689"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:07:48.550818 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.550676    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-workload-certs" pod "aff7037f-31de-4b57-a263-0229aa21b689" (UID: "aff7037f-31de-4b57-a263-0229aa21b689"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:07:48.551141 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.551118    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-credential-socket" pod "aff7037f-31de-4b57-a263-0229aa21b689" (UID: "aff7037f-31de-4b57-a263-0229aa21b689"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:07:48.551383 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.551230    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:07:48.551588 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.551398    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:07:48.551588 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.551420    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:07:48.553098 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.553069    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff7037f-31de-4b57-a263-0229aa21b689-istiod-ca-cert" pod "aff7037f-31de-4b57-a263-0229aa21b689" (UID: "aff7037f-31de-4b57-a263-0229aa21b689"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:07:48.553722 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.553689    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-istio-envoy" pod "aff7037f-31de-4b57-a263-0229aa21b689" (UID: "aff7037f-31de-4b57-a263-0229aa21b689"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:07:48.554051 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.553966    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff7037f-31de-4b57-a263-0229aa21b689-kube-api-access-pplxx" pod "aff7037f-31de-4b57-a263-0229aa21b689" (UID: "aff7037f-31de-4b57-a263-0229aa21b689"). InnerVolumeSpecName "kube-api-access-pplxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:07:48.554051 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.554014    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff7037f-31de-4b57-a263-0229aa21b689-istio-token" pod "aff7037f-31de-4b57-a263-0229aa21b689" (UID: "aff7037f-31de-4b57-a263-0229aa21b689"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:07:48.554217 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.554086    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aff7037f-31de-4b57-a263-0229aa21b689-istio-podinfo" pod "aff7037f-31de-4b57-a263-0229aa21b689" (UID: "aff7037f-31de-4b57-a263-0229aa21b689"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:07:48.655717 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.653556    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:07:48.655717 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.653593    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff7037f-31de-4b57-a263-0229aa21b689-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:07:48.655717 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.653609    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pplxx\" (UniqueName: \"kubernetes.io/projected/aff7037f-31de-4b57-a263-0229aa21b689-kube-api-access-pplxx\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:07:48.655717 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.653624    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff7037f-31de-4b57-a263-0229aa21b689-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:07:48.655717 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.653640    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff7037f-31de-4b57-a263-0229aa21b689-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:07:48.655717 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:48.653655    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff7037f-31de-4b57-a263-0229aa21b689-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:07:49.410799 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:49.410766    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"
Nov 03 00:07:49.411287 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:49.410774    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt" event={"ID":"aff7037f-31de-4b57-a263-0229aa21b689","Type":"ContainerDied","Data":"b87180287c29bfaac5abf43566aa77973038f8f0793d0cab58de6230b3cfa2d3"}
Nov 03 00:07:49.411287 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:49.410906    4294 scope.go:122] "RemoveContainer" containerID="9a49544380cba41eaa1a865959bcf5e224d97928daa9ac92df8d8bd1cf719ec0"
Nov 03 00:07:49.618776 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:49.618741    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"]
Nov 03 00:07:49.666532 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:49.666457    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-666a38-openshift-default-6475d6d845-f5xlt"]
Nov 03 00:07:50.445180 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:50.445145    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:50.548149 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:50.548116    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff7037f-31de-4b57-a263-0229aa21b689" path="/var/lib/kubelet/pods/aff7037f-31de-4b57-a263-0229aa21b689/volumes"
Nov 03 00:07:51.244572 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:51.244536    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:51.426088 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:51.426056    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:52.045555 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:52.045518    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:52.455608 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:52.455525    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:53.779120 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:53.779088    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:54.366654 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:54.366619    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:54.535747 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:54.535709    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:54.754921 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:54.754880    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:54.923125 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:54.923089    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:56.298712 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:56.298678    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:56.761159 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:56.761127    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:58.772620 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:07:58.772584    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:00.484246 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:00.484211    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:00.709392 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:00.709352    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:03.538584 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:03.538551    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:08.147362 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:08.147320    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:09.545968 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:09.545932    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:11.796290 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:08:11.796249    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:08:13.043166 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:13.043107    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:08:13.113921 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:13.113867    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:13.613003 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:13.612417    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb" podUID="aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:08:17.620117 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:17.620081    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:22.156526 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:22.156490    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:26.723535 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:26.723448    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:31.362004 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:31.361956    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:35.844273 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:35.844239    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:40.424557 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:40.424522    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:40.635168 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:40.635138    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:41.752944 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:08:41.752908    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:08:44.983766 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:44.983725    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:46.303227 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:46.303166    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:47.296868 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:47.296835    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:49.352462 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:49.352414    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:49.510205 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:49.510152    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:53.989704 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:53.989666    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:58.545344 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:08:58.545300    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:03.065404 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:03.065369    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:07.586669 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:07.586635    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:07.754546 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:07.754512    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:11.750766 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:09:11.750721    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:09:12.128443 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:12.128356    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:13.031284 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:13.031242    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:13.247200 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:13.247163    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:13.873546 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:13.873515    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:15.297996 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:15.297947    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:16.727528 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:16.727492    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:18.156042 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:18.156009    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:21.217573 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:21.217534    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:25.768946 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:25.768912    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:30.264072 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:30.264028    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:34.734646 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:34.734599    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:39.227433 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:39.227395    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:40.030808 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:40.030769    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:41.751049 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:09:41.751010    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:09:43.697627 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:43.697593    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:45.704056 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:45.704018    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:47.643797 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:47.643761    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:48.230720 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:48.230684    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:52.776268 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:52.776215    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:57.329151 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:09:57.329063    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:01.842176 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:01.842134    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:05.458537 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:05.458502    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:06.454821 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:06.454758    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:10.924465 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:10.924415    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:11.249807 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:11.249782    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:11.766805 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:10:11.766763    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:10:11.835021 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:11.834971    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:12.011224 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:12.011188    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:15.456083 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:15.456041    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:16.208241 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:16.208202    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:16.606749 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:16.606718    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:16.768922 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:16.768881    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:17.020209 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:17.020175    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:19.873861 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:19.873832    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:20.037583 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:20.037553    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:20.917425 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:20.917391    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:22.241075 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:22.241038    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:25.621517 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:25.621482    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:27.966910 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:27.966876    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:30.135578 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:30.135543    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:34.750959 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:34.750923    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:39.341689 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:39.341650    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:41.749075 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:10:41.749034    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:10:43.876632 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:43.876603    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:48.008021 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:48.007959    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:48.624242 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:48.624202    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:53.209788 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:53.209752    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:57.783066 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:10:57.783023    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:02.434419 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:02.434379    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:03.880914 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:03.880878    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:04.311388 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.311353    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"]
Nov 03 00:11:04.311632 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.311609    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47" podUID="f17fbbad-f397-418e-a74c-0ce90afd568b" containerName="discovery" containerID="cri-o://35b5e0f88323304565f30863ae220fd4f81da86007fed5236d572bcc93658894" gracePeriod=30
Nov 03 00:11:04.580500 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.580424    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:11:04.724302 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.724270    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-kubeconfig\") pod \"f17fbbad-f397-418e-a74c-0ce90afd568b\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") "
Nov 03 00:11:04.724515 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.724312    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-csr-ca-configmap\") pod \"f17fbbad-f397-418e-a74c-0ce90afd568b\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") "
Nov 03 00:11:04.724515 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.724332    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-token\" (UniqueName: \"kubernetes.io/projected/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-token\") pod \"f17fbbad-f397-418e-a74c-0ce90afd568b\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") "
Nov 03 00:11:04.724515 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.724408    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f17fbbad-f397-418e-a74c-0ce90afd568b-kube-api-access-f9fft\" (UniqueName: \"kubernetes.io/projected/f17fbbad-f397-418e-a74c-0ce90afd568b-kube-api-access-f9fft\") pod \"f17fbbad-f397-418e-a74c-0ce90afd568b\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") "
Nov 03 00:11:04.724515 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.724481    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-cacerts\" (UniqueName: \"kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-cacerts\") pod \"f17fbbad-f397-418e-a74c-0ce90afd568b\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") "
Nov 03 00:11:04.724757 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.724527    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f17fbbad-f397-418e-a74c-0ce90afd568b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f17fbbad-f397-418e-a74c-0ce90afd568b-local-certs\") pod \"f17fbbad-f397-418e-a74c-0ce90afd568b\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") "
Nov 03 00:11:04.724757 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.724651    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-csr-dns-cert\") pod \"f17fbbad-f397-418e-a74c-0ce90afd568b\" (UID: \"f17fbbad-f397-418e-a74c-0ce90afd568b\") "
Nov 03 00:11:04.724881 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.724768    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-csr-ca-configmap" pod "f17fbbad-f397-418e-a74c-0ce90afd568b" (UID: "f17fbbad-f397-418e-a74c-0ce90afd568b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:11:04.725271 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.725247    4294 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-csr-ca-configmap\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:11:04.727482 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.727453    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-cacerts" pod "f17fbbad-f397-418e-a74c-0ce90afd568b" (UID: "f17fbbad-f397-418e-a74c-0ce90afd568b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:11:04.727611 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.727551    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-token" pod "f17fbbad-f397-418e-a74c-0ce90afd568b" (UID: "f17fbbad-f397-418e-a74c-0ce90afd568b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:11:04.727754 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.727725    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f17fbbad-f397-418e-a74c-0ce90afd568b-kube-api-access-f9fft" pod "f17fbbad-f397-418e-a74c-0ce90afd568b" (UID: "f17fbbad-f397-418e-a74c-0ce90afd568b"). InnerVolumeSpecName "kube-api-access-f9fft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:11:04.727857 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.727796    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-csr-dns-cert" pod "f17fbbad-f397-418e-a74c-0ce90afd568b" (UID: "f17fbbad-f397-418e-a74c-0ce90afd568b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:11:04.727857 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.727810    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-kubeconfig" pod "f17fbbad-f397-418e-a74c-0ce90afd568b" (UID: "f17fbbad-f397-418e-a74c-0ce90afd568b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:11:04.728863 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.728842    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f17fbbad-f397-418e-a74c-0ce90afd568b-local-certs" pod "f17fbbad-f397-418e-a74c-0ce90afd568b" (UID: "f17fbbad-f397-418e-a74c-0ce90afd568b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:11:04.827037 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.826974    4294 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-kubeconfig\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:11:04.827037 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.827034    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:11:04.827037 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.827044    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9fft\" (UniqueName: \"kubernetes.io/projected/f17fbbad-f397-418e-a74c-0ce90afd568b-kube-api-access-f9fft\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:11:04.827330 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.827055    4294 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-cacerts\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:11:04.827330 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.827068    4294 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f17fbbad-f397-418e-a74c-0ce90afd568b-local-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:11:04.827330 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.827078    4294 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f17fbbad-f397-418e-a74c-0ce90afd568b-istio-csr-dns-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:11:04.907954 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.907862    4294 generic.go:356] "Generic (PLEG): container finished" podID="f17fbbad-f397-418e-a74c-0ce90afd568b" containerID="35b5e0f88323304565f30863ae220fd4f81da86007fed5236d572bcc93658894" exitCode=0
Nov 03 00:11:04.907954 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.907909    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"
Nov 03 00:11:04.908464 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.907947    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47" event={"ID":"f17fbbad-f397-418e-a74c-0ce90afd568b","Type":"ContainerDied","Data":"35b5e0f88323304565f30863ae220fd4f81da86007fed5236d572bcc93658894"}
Nov 03 00:11:04.908464 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.908010    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47" event={"ID":"f17fbbad-f397-418e-a74c-0ce90afd568b","Type":"ContainerDied","Data":"4fc3e748113639428da115e4e5002ca294bda778e44c113e484a85f7f529a593"}
Nov 03 00:11:04.908464 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.908033    4294 scope.go:122] "RemoveContainer" containerID="35b5e0f88323304565f30863ae220fd4f81da86007fed5236d572bcc93658894"
Nov 03 00:11:04.919584 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.919562    4294 scope.go:122] "RemoveContainer" containerID="35b5e0f88323304565f30863ae220fd4f81da86007fed5236d572bcc93658894"
Nov 03 00:11:04.919900 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:11:04.919879    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b5e0f88323304565f30863ae220fd4f81da86007fed5236d572bcc93658894\": container with ID starting with 35b5e0f88323304565f30863ae220fd4f81da86007fed5236d572bcc93658894 not found: ID does not exist" containerID="35b5e0f88323304565f30863ae220fd4f81da86007fed5236d572bcc93658894"
Nov 03 00:11:04.920004 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:04.919909    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b5e0f88323304565f30863ae220fd4f81da86007fed5236d572bcc93658894"} err="failed to get container status \"35b5e0f88323304565f30863ae220fd4f81da86007fed5236d572bcc93658894\": rpc error: code = NotFound desc = could not find container \"35b5e0f88323304565f30863ae220fd4f81da86007fed5236d572bcc93658894\": container with ID starting with 35b5e0f88323304565f30863ae220fd4f81da86007fed5236d572bcc93658894 not found: ID does not exist"
Nov 03 00:11:05.068148 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:05.068115    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"]
Nov 03 00:11:05.115997 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:05.115922    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rbl47"]
Nov 03 00:11:06.548559 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:06.548526    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f17fbbad-f397-418e-a74c-0ce90afd568b" path="/var/lib/kubelet/pods/f17fbbad-f397-418e-a74c-0ce90afd568b/volumes"
Nov 03 00:11:06.993423 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:06.993396    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:11.552889 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:11.552858    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:11.754203 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:11:11.754163    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:11:16.132343 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:16.132306    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:20.381421 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.381382    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"]
Nov 03 00:11:20.382071 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.381955    4294 cpu_manager.go:414] "containerMap: removing container" podUID="a6c764fb-09a2-4b5e-8588-b463f9487461" containerName="maas-api"
Nov 03 00:11:20.382071 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.381966    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6c764fb-09a2-4b5e-8588-b463f9487461" containerName="maas-api"
Nov 03 00:11:20.382071 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.382007    4294 cpu_manager.go:414] "containerMap: removing container" podUID="75f869b3-35bd-4168-bdd5-8a6334bc8e95" containerName="payload-pre-processing"
Nov 03 00:11:20.382071 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.382013    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f869b3-35bd-4168-bdd5-8a6334bc8e95" containerName="payload-pre-processing"
Nov 03 00:11:20.382071 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.382026    4294 cpu_manager.go:414] "containerMap: removing container" podUID="aff7037f-31de-4b57-a263-0229aa21b689" containerName="istio-proxy"
Nov 03 00:11:20.382071 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.382030    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff7037f-31de-4b57-a263-0229aa21b689" containerName="istio-proxy"
Nov 03 00:11:20.382071 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.382042    4294 cpu_manager.go:414] "containerMap: removing container" podUID="f17fbbad-f397-418e-a74c-0ce90afd568b" containerName="discovery"
Nov 03 00:11:20.382071 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.382046    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f17fbbad-f397-418e-a74c-0ce90afd568b" containerName="discovery"
Nov 03 00:11:20.382530 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.382167    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f869b3-35bd-4168-bdd5-8a6334bc8e95" containerName="payload-pre-processing"
Nov 03 00:11:20.382530 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.382177    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff7037f-31de-4b57-a263-0229aa21b689" containerName="istio-proxy"
Nov 03 00:11:20.382530 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.382183    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="f17fbbad-f397-418e-a74c-0ce90afd568b" containerName="discovery"
Nov 03 00:11:20.382530 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.382193    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6c764fb-09a2-4b5e-8588-b463f9487461" containerName="maas-api"
Nov 03 00:11:20.386793 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.386772    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"
Nov 03 00:11:20.409153 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.409118    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7940146-b317-4fa3-9193-0aad4dd2a294-openshift-service-ca\") pod \"maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9\" (UID: \"b7940146-b317-4fa3-9193-0aad4dd2a294\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"
Nov 03 00:11:20.409305 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.409239    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bqbl\" (UniqueName: \"kubernetes.io/projected/b7940146-b317-4fa3-9193-0aad4dd2a294-kube-api-access-5bqbl\") pod \"maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9\" (UID: \"b7940146-b317-4fa3-9193-0aad4dd2a294\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"
Nov 03 00:11:20.409413 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.409393    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7940146-b317-4fa3-9193-0aad4dd2a294-maas-api-tls\") pod \"maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9\" (UID: \"b7940146-b317-4fa3-9193-0aad4dd2a294\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"
Nov 03 00:11:20.409476 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.409463    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7940146-b317-4fa3-9193-0aad4dd2a294-metrics-certs\") pod \"maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9\" (UID: \"b7940146-b317-4fa3-9193-0aad4dd2a294\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"
Nov 03 00:11:20.439729 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.439686    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"]
Nov 03 00:11:20.481824 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.481784    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1f668443-serving-cert\""
Nov 03 00:11:20.510910 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.510868    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7940146-b317-4fa3-9193-0aad4dd2a294-openshift-service-ca\") pod \"maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9\" (UID: \"b7940146-b317-4fa3-9193-0aad4dd2a294\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"
Nov 03 00:11:20.511184 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.510961    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bqbl\" (UniqueName: \"kubernetes.io/projected/b7940146-b317-4fa3-9193-0aad4dd2a294-kube-api-access-5bqbl\") pod \"maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9\" (UID: \"b7940146-b317-4fa3-9193-0aad4dd2a294\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"
Nov 03 00:11:20.511184 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.511118    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7940146-b317-4fa3-9193-0aad4dd2a294-maas-api-tls\") pod \"maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9\" (UID: \"b7940146-b317-4fa3-9193-0aad4dd2a294\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"
Nov 03 00:11:20.511332 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.511183    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7940146-b317-4fa3-9193-0aad4dd2a294-metrics-certs\") pod \"maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9\" (UID: \"b7940146-b317-4fa3-9193-0aad4dd2a294\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"
Nov 03 00:11:20.511399 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:11:20.511330    4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-serving-cert: secret "maas-api-e2e-mt-1f668443-serving-cert" not found
Nov 03 00:11:20.511457 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:11:20.511408    4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7940146-b317-4fa3-9193-0aad4dd2a294-maas-api-tls podName:b7940146-b317-4fa3-9193-0aad4dd2a294 nodeName:}" failed. No retries permitted until 2026-11-03 00:11:21.011389382 +0000 UTC m=+4720.579744150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7940146-b317-4fa3-9193-0aad4dd2a294-maas-api-tls") pod "maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9" (UID: "b7940146-b317-4fa3-9193-0aad4dd2a294") : secret "maas-api-e2e-mt-1f668443-serving-cert" not found
Nov 03 00:11:20.511785 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.511755    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7940146-b317-4fa3-9193-0aad4dd2a294-openshift-service-ca\") pod \"maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9\" (UID: \"b7940146-b317-4fa3-9193-0aad4dd2a294\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"
Nov 03 00:11:20.514243 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.514217    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7940146-b317-4fa3-9193-0aad4dd2a294-metrics-certs\") pod \"maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9\" (UID: \"b7940146-b317-4fa3-9193-0aad4dd2a294\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"
Nov 03 00:11:20.565902 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.565864    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bqbl\" (UniqueName: \"kubernetes.io/projected/b7940146-b317-4fa3-9193-0aad4dd2a294-kube-api-access-5bqbl\") pod \"maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9\" (UID: \"b7940146-b317-4fa3-9193-0aad4dd2a294\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"
Nov 03 00:11:20.589437 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.589405    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:20.786454 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.786404    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:20.986647 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:20.986616    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:21.017668 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:21.017617    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7940146-b317-4fa3-9193-0aad4dd2a294-maas-api-tls\") pod \"maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9\" (UID: \"b7940146-b317-4fa3-9193-0aad4dd2a294\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"
Nov 03 00:11:21.020307 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:21.020274    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7940146-b317-4fa3-9193-0aad4dd2a294-maas-api-tls\") pod \"maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9\" (UID: \"b7940146-b317-4fa3-9193-0aad4dd2a294\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"
Nov 03 00:11:21.301686 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:21.301653    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"
Nov 03 00:11:21.671699 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:21.671669    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"]
Nov 03 00:11:21.672388 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:11:21.672359    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7940146_b317_4fa3_9193_0aad4dd2a294.slice/crio-05707a48547a2c447f1056a9fa41f1dc61eb71af28b8e0b6914731519becade3 WatchSource:0}: Error finding container 05707a48547a2c447f1056a9fa41f1dc61eb71af28b8e0b6914731519becade3: Status 404 returned error can't find the container with id 05707a48547a2c447f1056a9fa41f1dc61eb71af28b8e0b6914731519becade3
Nov 03 00:11:21.678157 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:21.678137    4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:11:22.018012 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:22.017958    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9" event={"ID":"b7940146-b317-4fa3-9193-0aad4dd2a294","Type":"ContainerStarted","Data":"05707a48547a2c447f1056a9fa41f1dc61eb71af28b8e0b6914731519becade3"}
Nov 03 00:11:23.026049 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:23.026012    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9" event={"ID":"b7940146-b317-4fa3-9193-0aad4dd2a294","Type":"ContainerStarted","Data":"8444d3c980a64f78f3991a4646cdd9cc4dd93c082874a79f6f2aaec85431870c"}
Nov 03 00:11:23.026473 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:23.026151    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"
Nov 03 00:11:23.135650 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:23.135594    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9" podStartSLOduration=2.128815856 podStartE2EDuration="3.135581289s" totalImagesPullingTime="1.006765433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:23.134787296 +0000 UTC m=+4722.703142093" watchObservedRunningTime="2026-11-03 00:11:23.135581289 +0000 UTC m=+4722.703936075"
Nov 03 00:11:25.344187 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:25.344102    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:29.041862 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:29.041830    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"
Nov 03 00:11:29.930169 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:29.930121    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:34.545784 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:34.545726    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:38.924654 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:38.924621    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:41.752323 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:11:41.752288    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:11:43.435102 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:43.435068    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:47.927966 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:47.927923    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:48.318143 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:48.318108    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:52.394323 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:52.394292    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:52.745173 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:52.745132    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:52.925163 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:52.925108    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:53.292372 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:53.292339    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:56.898085 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:56.898050    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:59.152198 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:59.152166    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:59.313683 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:11:59.313650    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:01.453944 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:01.453907    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:04.208829 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:04.208791    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"]
Nov 03 00:12:04.256246 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:04.256218    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"]
Nov 03 00:12:04.406366 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:04.406339    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:05.938447 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:05.938418    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:09.914330 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:09.914290    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"]
Nov 03 00:12:09.918858 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:09.918836    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:09.971243 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:09.971209    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"]
Nov 03 00:12:09.988731 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:09.988708    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-b4d53e\"/\"openshift-service-ca.crt\""
Nov 03 00:12:09.988914 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:09.988899    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-b4d53e\"/\"kube-root-ca.crt\""
Nov 03 00:12:10.017960 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.017925    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-tmp-dir\") pod \"sub-test-model-b4d53e-kserve-685dcdff5f-whtc7\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:10.018162 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.018011    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/989580e5-edec-4c8b-97b8-c9e105dc3306-tls-certs\") pod \"sub-test-model-b4d53e-kserve-685dcdff5f-whtc7\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:10.018162 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.018106    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqzvt\" (UniqueName: \"kubernetes.io/projected/989580e5-edec-4c8b-97b8-c9e105dc3306-kube-api-access-bqzvt\") pod \"sub-test-model-b4d53e-kserve-685dcdff5f-whtc7\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:10.018274 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.018257    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-model-cache\") pod \"sub-test-model-b4d53e-kserve-685dcdff5f-whtc7\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:10.018331 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.018316    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-dshm\") pod \"sub-test-model-b4d53e-kserve-685dcdff5f-whtc7\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:10.018430 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.018414    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-home\") pod \"sub-test-model-b4d53e-kserve-685dcdff5f-whtc7\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:10.061634 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.061591    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-b4d53e\"/\"default-dockercfg-tvtlq\""
Nov 03 00:12:10.061634 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.061591    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-b4d53e\"/\"sub-test-model-b4d53e-kserve-self-signed-certs\""
Nov 03 00:12:10.088934 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.088905    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:10.119999 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.119959    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/989580e5-edec-4c8b-97b8-c9e105dc3306-tls-certs\") pod \"sub-test-model-b4d53e-kserve-685dcdff5f-whtc7\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:10.120168 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.120034    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqzvt\" (UniqueName: \"kubernetes.io/projected/989580e5-edec-4c8b-97b8-c9e105dc3306-kube-api-access-bqzvt\") pod \"sub-test-model-b4d53e-kserve-685dcdff5f-whtc7\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:10.120228 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.120176    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-model-cache\") pod \"sub-test-model-b4d53e-kserve-685dcdff5f-whtc7\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:10.120288 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.120248    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-dshm\") pod \"sub-test-model-b4d53e-kserve-685dcdff5f-whtc7\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:10.120346 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.120299    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-home\") pod \"sub-test-model-b4d53e-kserve-685dcdff5f-whtc7\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:10.120404 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.120381    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-tmp-dir\") pod \"sub-test-model-b4d53e-kserve-685dcdff5f-whtc7\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:10.120628 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.120597    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-model-cache\") pod \"sub-test-model-b4d53e-kserve-685dcdff5f-whtc7\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:10.120701 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.120678    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-home\") pod \"sub-test-model-b4d53e-kserve-685dcdff5f-whtc7\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:10.120742 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.120726    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-tmp-dir\") pod \"sub-test-model-b4d53e-kserve-685dcdff5f-whtc7\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:10.122287 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.122268    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-dshm\") pod \"sub-test-model-b4d53e-kserve-685dcdff5f-whtc7\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:10.122562 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.122546    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/989580e5-edec-4c8b-97b8-c9e105dc3306-tls-certs\") pod \"sub-test-model-b4d53e-kserve-685dcdff5f-whtc7\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:10.174694 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.174620    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqzvt\" (UniqueName: \"kubernetes.io/projected/989580e5-edec-4c8b-97b8-c9e105dc3306-kube-api-access-bqzvt\") pod \"sub-test-model-b4d53e-kserve-685dcdff5f-whtc7\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") " pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:10.231772 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.231738    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:10.481435 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.481344    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:10.603004 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.602944    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"]
Nov 03 00:12:10.606800 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:12:10.606767    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod989580e5_edec_4c8b_97b8_c9e105dc3306.slice/crio-d7191343daf6a03fe2f7f1c65cb738b7916e4c76df28b17a4cd9fc88da56691e WatchSource:0}: Error finding container d7191343daf6a03fe2f7f1c65cb738b7916e4c76df28b17a4cd9fc88da56691e: Status 404 returned error can't find the container with id d7191343daf6a03fe2f7f1c65cb738b7916e4c76df28b17a4cd9fc88da56691e
Nov 03 00:12:10.647355 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:10.647321    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:11.330255 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:11.330223    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7" event={"ID":"989580e5-edec-4c8b-97b8-c9e105dc3306","Type":"ContainerStarted","Data":"7e00fcd6b3b6c9cf08739e07536012d1c79b0c1f771b91b7d63381005dee2323"}
Nov 03 00:12:11.330631 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:11.330263    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7" event={"ID":"989580e5-edec-4c8b-97b8-c9e105dc3306","Type":"ContainerStarted","Data":"d7191343daf6a03fe2f7f1c65cb738b7916e4c76df28b17a4cd9fc88da56691e"}
Nov 03 00:12:11.440459 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:11.440410    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7" podStartSLOduration=2.44039555 podStartE2EDuration="2.44039555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:12:11.438720227 +0000 UTC m=+4771.007075027" watchObservedRunningTime="2026-11-03 00:12:11.44039555 +0000 UTC m=+4771.008750364"
Nov 03 00:12:11.750431 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:12:11.750395    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:12:12.232285 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:12.232198    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:15.147860 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:15.147817    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:16.939122 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:16.939072    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:18.069133 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:18.069099    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:18.231550 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:18.231518    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:19.229844 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.229792    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main" containerID="cri-o://f94a39b8d0087ee97f2c07169ce6030c74fa0b8debfb0e0e08bb54ff3069c164" gracePeriod=45
Nov 03 00:12:19.276027 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.275972    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main" containerID="cri-o://1e28a71498a14d121ff7e9e919439fdd933e24883f0424eb7706ba2c79084473" gracePeriod=45
Nov 03 00:12:19.385485 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.385458    4294 generic.go:356] "Generic (PLEG): container finished" podID="67d1920b-7d2f-4946-9378-efa0bd270339" containerID="f94a39b8d0087ee97f2c07169ce6030c74fa0b8debfb0e0e08bb54ff3069c164" exitCode=0
Nov 03 00:12:19.385627 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.385538    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" event={"ID":"67d1920b-7d2f-4946-9378-efa0bd270339","Type":"ContainerDied","Data":"f94a39b8d0087ee97f2c07169ce6030c74fa0b8debfb0e0e08bb54ff3069c164"}
Nov 03 00:12:19.545347 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.545320    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 03 00:12:19.572306 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.572283    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 03 00:12:19.650742 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.650709    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-model-cache\") pod \"67d1920b-7d2f-4946-9378-efa0bd270339\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") "
Nov 03 00:12:19.650949 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.650893    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d40c2149-5d44-44e5-b271-6ebd82197e22-tls-certs\" (UniqueName: \"kubernetes.io/secret/d40c2149-5d44-44e5-b271-6ebd82197e22-tls-certs\") pod \"d40c2149-5d44-44e5-b271-6ebd82197e22\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") "
Nov 03 00:12:19.650949 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.650931    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-dshm\") pod \"d40c2149-5d44-44e5-b271-6ebd82197e22\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") "
Nov 03 00:12:19.651087 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.650959    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-tmp-dir\") pod \"d40c2149-5d44-44e5-b271-6ebd82197e22\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") "
Nov 03 00:12:19.651087 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.651017    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d1920b-7d2f-4946-9378-efa0bd270339-kube-api-access-zkj85\" (UniqueName: \"kubernetes.io/projected/67d1920b-7d2f-4946-9378-efa0bd270339-kube-api-access-zkj85\") pod \"67d1920b-7d2f-4946-9378-efa0bd270339\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") "
Nov 03 00:12:19.651087 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.651042    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d40c2149-5d44-44e5-b271-6ebd82197e22-kube-api-access-bj5zb\" (UniqueName: \"kubernetes.io/projected/d40c2149-5d44-44e5-b271-6ebd82197e22-kube-api-access-bj5zb\") pod \"d40c2149-5d44-44e5-b271-6ebd82197e22\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") "
Nov 03 00:12:19.651087 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.650955    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-model-cache" pod "67d1920b-7d2f-4946-9378-efa0bd270339" (UID: "67d1920b-7d2f-4946-9378-efa0bd270339"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:12:19.651087 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.651066    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-home\" (UniqueName: \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-home\") pod \"67d1920b-7d2f-4946-9378-efa0bd270339\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") "
Nov 03 00:12:19.651353 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.651123    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67d1920b-7d2f-4946-9378-efa0bd270339-tls-certs\" (UniqueName: \"kubernetes.io/secret/67d1920b-7d2f-4946-9378-efa0bd270339-tls-certs\") pod \"67d1920b-7d2f-4946-9378-efa0bd270339\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") "
Nov 03 00:12:19.651353 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.651158    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-model-cache\") pod \"d40c2149-5d44-44e5-b271-6ebd82197e22\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") "
Nov 03 00:12:19.651353 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.651186    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-home\" (UniqueName: \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-home\") pod \"d40c2149-5d44-44e5-b271-6ebd82197e22\" (UID: \"d40c2149-5d44-44e5-b271-6ebd82197e22\") "
Nov 03 00:12:19.651353 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.651201    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-tmp-dir" pod "d40c2149-5d44-44e5-b271-6ebd82197e22" (UID: "d40c2149-5d44-44e5-b271-6ebd82197e22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:12:19.651353 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.651237    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-dshm\" (UniqueName: \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-dshm\") pod \"67d1920b-7d2f-4946-9378-efa0bd270339\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") "
Nov 03 00:12:19.651353 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.651278    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-tmp-dir\") pod \"67d1920b-7d2f-4946-9378-efa0bd270339\" (UID: \"67d1920b-7d2f-4946-9378-efa0bd270339\") "
Nov 03 00:12:19.651680 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.651553    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-home" pod "67d1920b-7d2f-4946-9378-efa0bd270339" (UID: "67d1920b-7d2f-4946-9378-efa0bd270339"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:12:19.651680 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.651587    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-home" pod "d40c2149-5d44-44e5-b271-6ebd82197e22" (UID: "d40c2149-5d44-44e5-b271-6ebd82197e22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:12:19.652117 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.652091    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:19.652230 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.652121    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:19.652230 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.652136    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:19.652230 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.652150    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:19.652434 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.652363    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-tmp-dir" pod "67d1920b-7d2f-4946-9378-efa0bd270339" (UID: "67d1920b-7d2f-4946-9378-efa0bd270339"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:12:19.652434 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.652392    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-model-cache" pod "d40c2149-5d44-44e5-b271-6ebd82197e22" (UID: "d40c2149-5d44-44e5-b271-6ebd82197e22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:12:19.653803 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.653773    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d40c2149-5d44-44e5-b271-6ebd82197e22-tls-certs" pod "d40c2149-5d44-44e5-b271-6ebd82197e22" (UID: "d40c2149-5d44-44e5-b271-6ebd82197e22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:12:19.653997 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.653959    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-dshm" pod "67d1920b-7d2f-4946-9378-efa0bd270339" (UID: "67d1920b-7d2f-4946-9378-efa0bd270339"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:12:19.654106 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.653968    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67d1920b-7d2f-4946-9378-efa0bd270339-tls-certs" pod "67d1920b-7d2f-4946-9378-efa0bd270339" (UID: "67d1920b-7d2f-4946-9378-efa0bd270339"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:12:19.654215 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.654193    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40c2149-5d44-44e5-b271-6ebd82197e22-kube-api-access-bj5zb" pod "d40c2149-5d44-44e5-b271-6ebd82197e22" (UID: "d40c2149-5d44-44e5-b271-6ebd82197e22"). InnerVolumeSpecName "kube-api-access-bj5zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:12:19.655282 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.655258    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d1920b-7d2f-4946-9378-efa0bd270339-kube-api-access-zkj85" pod "67d1920b-7d2f-4946-9378-efa0bd270339" (UID: "67d1920b-7d2f-4946-9378-efa0bd270339"). InnerVolumeSpecName "kube-api-access-zkj85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:12:19.655467 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.655333    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-dshm" pod "d40c2149-5d44-44e5-b271-6ebd82197e22" (UID: "d40c2149-5d44-44e5-b271-6ebd82197e22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:12:19.753553 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.753519    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d40c2149-5d44-44e5-b271-6ebd82197e22-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:19.753553 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.753549    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:19.753768 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.753562    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkj85\" (UniqueName: \"kubernetes.io/projected/67d1920b-7d2f-4946-9378-efa0bd270339-kube-api-access-zkj85\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:19.753768 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.753577    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj5zb\" (UniqueName: \"kubernetes.io/projected/d40c2149-5d44-44e5-b271-6ebd82197e22-kube-api-access-bj5zb\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:19.753768 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.753589    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d1920b-7d2f-4946-9378-efa0bd270339-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:19.753768 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.753601    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d40c2149-5d44-44e5-b271-6ebd82197e22-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:19.753768 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.753612    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:19.753768 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:19.753625    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d1920b-7d2f-4946-9378-efa0bd270339-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:20.120442 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:20.120404    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:20.232857 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:20.232821    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:20.245217 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:20.245188    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:20.393520 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:20.393432    4294 generic.go:356] "Generic (PLEG): container finished" podID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerID="1e28a71498a14d121ff7e9e919439fdd933e24883f0424eb7706ba2c79084473" exitCode=0
Nov 03 00:12:20.393687 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:20.393517    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"
Nov 03 00:12:20.393687 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:20.393524    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" event={"ID":"d40c2149-5d44-44e5-b271-6ebd82197e22","Type":"ContainerDied","Data":"1e28a71498a14d121ff7e9e919439fdd933e24883f0424eb7706ba2c79084473"}
Nov 03 00:12:20.393687 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:20.393560    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz" event={"ID":"d40c2149-5d44-44e5-b271-6ebd82197e22","Type":"ContainerDied","Data":"2ac68cc06889ad718809d202994bec2cc62a6040ef76e6cefd94a27aeece314b"}
Nov 03 00:12:20.393687 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:20.393580    4294 scope.go:122] "RemoveContainer" containerID="1e28a71498a14d121ff7e9e919439fdd933e24883f0424eb7706ba2c79084473"
Nov 03 00:12:20.394948 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:20.394896    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"
Nov 03 00:12:20.394948 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:20.394936    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf" event={"ID":"67d1920b-7d2f-4946-9378-efa0bd270339","Type":"ContainerDied","Data":"e0ec62f7ac03ab07e3f84fce96ffa53d240c20c208de76f02cd8a0689a8ed159"}
Nov 03 00:12:20.404076 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:20.404060    4294 scope.go:122] "RemoveContainer" containerID="1e28a71498a14d121ff7e9e919439fdd933e24883f0424eb7706ba2c79084473"
Nov 03 00:12:20.404380 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:12:20.404363    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e28a71498a14d121ff7e9e919439fdd933e24883f0424eb7706ba2c79084473\": container with ID starting with 1e28a71498a14d121ff7e9e919439fdd933e24883f0424eb7706ba2c79084473 not found: ID does not exist" containerID="1e28a71498a14d121ff7e9e919439fdd933e24883f0424eb7706ba2c79084473"
Nov 03 00:12:20.404480 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:20.404390    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e28a71498a14d121ff7e9e919439fdd933e24883f0424eb7706ba2c79084473"} err="failed to get container status \"1e28a71498a14d121ff7e9e919439fdd933e24883f0424eb7706ba2c79084473\": rpc error: code = NotFound desc = could not find container \"1e28a71498a14d121ff7e9e919439fdd933e24883f0424eb7706ba2c79084473\": container with ID starting with 1e28a71498a14d121ff7e9e919439fdd933e24883f0424eb7706ba2c79084473 not found: ID does not exist"
Nov 03 00:12:20.404480 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:20.404410    4294 scope.go:122] "RemoveContainer" containerID="f94a39b8d0087ee97f2c07169ce6030c74fa0b8debfb0e0e08bb54ff3069c164"
Nov 03 00:12:20.410359 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:20.410332    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:12:20.550393 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:20.550365    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"]
Nov 03 00:12:20.553269 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:20.553250    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/distinct-w0-59f97a-kserve-8f86ccf58-h76gz"]
Nov 03 00:12:20.655750 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:20.655646    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"]
Nov 03 00:12:20.659063 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:20.659041    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-59f97a/distinct-2-w0-59f97a-kserve-5c87465b69-ptzpf"]
Nov 03 00:12:22.426003 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:22.425606    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:22.551469 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:22.551429    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" path="/var/lib/kubelet/pods/67d1920b-7d2f-4946-9378-efa0bd270339/volumes"
Nov 03 00:12:22.552927 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:22.552104    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" path="/var/lib/kubelet/pods/d40c2149-5d44-44e5-b271-6ebd82197e22/volumes"
Nov 03 00:12:23.218925 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.218886    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"]
Nov 03 00:12:23.219557 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.219538    4294 cpu_manager.go:414] "containerMap: removing container" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main"
Nov 03 00:12:23.219557 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.219556    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main"
Nov 03 00:12:23.219727 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.219575    4294 cpu_manager.go:414] "containerMap: removing container" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main"
Nov 03 00:12:23.219727 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.219580    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main"
Nov 03 00:12:23.219727 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.219677    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d1920b-7d2f-4946-9378-efa0bd270339" containerName="main"
Nov 03 00:12:23.219727 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.219688    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d40c2149-5d44-44e5-b271-6ebd82197e22" containerName="main"
Nov 03 00:12:23.224570 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.224553    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:23.279290 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.279257    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"]
Nov 03 00:12:23.314774 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.314061    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-home\") pod \"sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:23.314774 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.314160    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s575q\" (UniqueName: \"kubernetes.io/projected/785b24ed-8d15-410f-b7d4-02549e6e2201-kube-api-access-s575q\") pod \"sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:23.314774 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.314201    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-tmp-dir\") pod \"sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:23.314774 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.314329    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/785b24ed-8d15-410f-b7d4-02549e6e2201-tls-certs\") pod \"sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:23.314774 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.314428    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-model-cache\") pod \"sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:23.314774 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.314584    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-dshm\") pod \"sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:23.323207 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.323114    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7ca60f\"/\"kube-root-ca.crt\""
Nov 03 00:12:23.323392 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.323276    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7ca60f\"/\"default-dockercfg-j88dc\""
Nov 03 00:12:23.323392 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.323369    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7ca60f\"/\"sub-test-model-7ca60f-kserve-self-signed-certs\""
Nov 03 00:12:23.323526 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.323402    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7ca60f\"/\"openshift-service-ca.crt\""
Nov 03 00:12:23.333809 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.333779    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:23.415962 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.415929    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-home\") pod \"sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:23.416201 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.416014    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s575q\" (UniqueName: \"kubernetes.io/projected/785b24ed-8d15-410f-b7d4-02549e6e2201-kube-api-access-s575q\") pod \"sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:23.416201 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.416052    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-tmp-dir\") pod \"sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:23.416201 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.416164    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/785b24ed-8d15-410f-b7d4-02549e6e2201-tls-certs\") pod \"sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:23.416402 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.416200    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-model-cache\") pod \"sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:23.416402 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.416318    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-dshm\") pod \"sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:23.416402 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.416385    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-home\") pod \"sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:23.417029 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.416656    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-tmp-dir\") pod \"sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:23.417029 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.416945    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-model-cache\") pod \"sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:23.420254 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.420229    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-dshm\") pod \"sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:23.420636 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.420617    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/785b24ed-8d15-410f-b7d4-02549e6e2201-tls-certs\") pod \"sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:23.474596 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.474511    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s575q\" (UniqueName: \"kubernetes.io/projected/785b24ed-8d15-410f-b7d4-02549e6e2201-kube-api-access-s575q\") pod \"sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") " pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:23.536880 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.536840    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:23.907347 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:23.907305    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"]
Nov 03 00:12:23.907582 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:12:23.907558    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod785b24ed_8d15_410f_b7d4_02549e6e2201.slice/crio-a8f41df67a993e3f53e8f6c22863a50cff6b68d8d1a691ca225aab6ad29bf4d1 WatchSource:0}: Error finding container a8f41df67a993e3f53e8f6c22863a50cff6b68d8d1a691ca225aab6ad29bf4d1: Status 404 returned error can't find the container with id a8f41df67a993e3f53e8f6c22863a50cff6b68d8d1a691ca225aab6ad29bf4d1
Nov 03 00:12:24.444655 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:24.444613    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48" event={"ID":"785b24ed-8d15-410f-b7d4-02549e6e2201","Type":"ContainerStarted","Data":"a814ac8af58629c4a3c2c4002498978b5ca801ee330db383be5afe2c5fd8ffe0"}
Nov 03 00:12:24.444866 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:24.444663    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48" event={"ID":"785b24ed-8d15-410f-b7d4-02549e6e2201","Type":"ContainerStarted","Data":"a8f41df67a993e3f53e8f6c22863a50cff6b68d8d1a691ca225aab6ad29bf4d1"}
Nov 03 00:12:24.537436 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:24.537404    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:24.555583 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:24.555529    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48" podStartSLOduration=1.555515652 podStartE2EDuration="1.555515652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:12:24.553700307 +0000 UTC m=+4784.122055088" watchObservedRunningTime="2026-11-03 00:12:24.555515652 +0000 UTC m=+4784.123870438"
Nov 03 00:12:24.854721 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:24.854685    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:29.437139 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:29.437097    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:33.537840 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:33.537807    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:33.549938 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:33.549911    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:34.248183 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:34.248132    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:34.520160 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:34.520087    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:12:37.032038 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:37.031972    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:39.076639 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:39.076595    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:39.488307 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:39.488229    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:40.306640 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:40.306607    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:40.510930 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:40.510880    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:40.879111 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:40.879077    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:41.751032 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:12:41.750993    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:12:42.084221 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:42.084139    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:42.245650 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:42.245615    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:42.416591 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:42.416496    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:43.762684 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:43.762650    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:46.496169 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:46.496136    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:47.725598 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:47.725545    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:47.963906 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:47.963866    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:48.663962 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:48.663929    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:49.353788 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:49.353757    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:51.921915 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:51.921873    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:53.841554 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:53.841479    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:54.123872 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:54.123047    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"]
Nov 03 00:12:54.123872 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:54.123311    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg" podUID="b007881f-8d1a-487c-ab6f-0ad740dd047a" containerName="istio-proxy" containerID="cri-o://02a9e8323a63b9d2981e06b1c6a74041cb12c22839f9eb18785395a91bb38b98" gracePeriod=30
Nov 03 00:12:54.137698 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:54.137666    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:54.311175 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:54.311138    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:58.944314 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:58.944278    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:59.402441 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.402416    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 03 00:12:59.454427 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.454396    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b007881f-8d1a-487c-ab6f-0ad740dd047a-kube-api-access-5gzb6\" (UniqueName: \"kubernetes.io/projected/b007881f-8d1a-487c-ab6f-0ad740dd047a-kube-api-access-5gzb6\") pod \"b007881f-8d1a-487c-ab6f-0ad740dd047a\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") "
Nov 03 00:12:59.454625 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.454443    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-credential-socket\") pod \"b007881f-8d1a-487c-ab6f-0ad740dd047a\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") "
Nov 03 00:12:59.454625 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.454585    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-data\") pod \"b007881f-8d1a-487c-ab6f-0ad740dd047a\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") "
Nov 03 00:12:59.454625 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.454623    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-token\" (UniqueName: \"kubernetes.io/projected/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-token\") pod \"b007881f-8d1a-487c-ab6f-0ad740dd047a\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") "
Nov 03 00:12:59.454803 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.454677    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-workload-certs\") pod \"b007881f-8d1a-487c-ab6f-0ad740dd047a\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") "
Nov 03 00:12:59.454803 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.454712    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-envoy\") pod \"b007881f-8d1a-487c-ab6f-0ad740dd047a\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") "
Nov 03 00:12:59.454803 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.454742    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-workload-socket\") pod \"b007881f-8d1a-487c-ab6f-0ad740dd047a\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") "
Nov 03 00:12:59.454803 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.454790    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-podinfo\") pod \"b007881f-8d1a-487c-ab6f-0ad740dd047a\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") "
Nov 03 00:12:59.455038 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.454818    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b007881f-8d1a-487c-ab6f-0ad740dd047a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b007881f-8d1a-487c-ab6f-0ad740dd047a-istiod-ca-cert\") pod \"b007881f-8d1a-487c-ab6f-0ad740dd047a\" (UID: \"b007881f-8d1a-487c-ab6f-0ad740dd047a\") "
Nov 03 00:12:59.455038 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.454820    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-credential-socket" pod "b007881f-8d1a-487c-ab6f-0ad740dd047a" (UID: "b007881f-8d1a-487c-ab6f-0ad740dd047a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:12:59.455038 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.454876    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-data" pod "b007881f-8d1a-487c-ab6f-0ad740dd047a" (UID: "b007881f-8d1a-487c-ab6f-0ad740dd047a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:12:59.455202 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.455102    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-workload-certs" pod "b007881f-8d1a-487c-ab6f-0ad740dd047a" (UID: "b007881f-8d1a-487c-ab6f-0ad740dd047a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:12:59.455202 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.455112    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-workload-socket" pod "b007881f-8d1a-487c-ab6f-0ad740dd047a" (UID: "b007881f-8d1a-487c-ab6f-0ad740dd047a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:12:59.455488 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.455385    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:59.455488 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.455419    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:59.455488 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.455435    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:59.455488 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.455435    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b007881f-8d1a-487c-ab6f-0ad740dd047a-istiod-ca-cert" pod "b007881f-8d1a-487c-ab6f-0ad740dd047a" (UID: "b007881f-8d1a-487c-ab6f-0ad740dd047a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:12:59.455488 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.455449    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:59.457339 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.457313    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-token" pod "b007881f-8d1a-487c-ab6f-0ad740dd047a" (UID: "b007881f-8d1a-487c-ab6f-0ad740dd047a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:12:59.457446 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.457397    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b007881f-8d1a-487c-ab6f-0ad740dd047a-kube-api-access-5gzb6" pod "b007881f-8d1a-487c-ab6f-0ad740dd047a" (UID: "b007881f-8d1a-487c-ab6f-0ad740dd047a"). InnerVolumeSpecName "kube-api-access-5gzb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:12:59.457446 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.457423    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-envoy" pod "b007881f-8d1a-487c-ab6f-0ad740dd047a" (UID: "b007881f-8d1a-487c-ab6f-0ad740dd047a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:12:59.457715 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.457693    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-podinfo" pod "b007881f-8d1a-487c-ab6f-0ad740dd047a" (UID: "b007881f-8d1a-487c-ab6f-0ad740dd047a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:12:59.556233 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.556141    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gzb6\" (UniqueName: \"kubernetes.io/projected/b007881f-8d1a-487c-ab6f-0ad740dd047a-kube-api-access-5gzb6\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:59.556233 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.556178    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:59.556233 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.556195    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:59.556233 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.556210    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b007881f-8d1a-487c-ab6f-0ad740dd047a-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:59.556233 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.556223    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b007881f-8d1a-487c-ab6f-0ad740dd047a-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:12:59.662769 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.662729    4294 generic.go:356] "Generic (PLEG): container finished" podID="b007881f-8d1a-487c-ab6f-0ad740dd047a" containerID="02a9e8323a63b9d2981e06b1c6a74041cb12c22839f9eb18785395a91bb38b98" exitCode=0
Nov 03 00:12:59.662970 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.662781    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"
Nov 03 00:12:59.662970 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.662819    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg" event={"ID":"b007881f-8d1a-487c-ab6f-0ad740dd047a","Type":"ContainerDied","Data":"02a9e8323a63b9d2981e06b1c6a74041cb12c22839f9eb18785395a91bb38b98"}
Nov 03 00:12:59.662970 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.662883    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg" event={"ID":"b007881f-8d1a-487c-ab6f-0ad740dd047a","Type":"ContainerDied","Data":"37471dafc7bac8f629b84e3271b98b07214a1174637882e27267900c6d6ffa02"}
Nov 03 00:12:59.662970 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.662912    4294 scope.go:122] "RemoveContainer" containerID="02a9e8323a63b9d2981e06b1c6a74041cb12c22839f9eb18785395a91bb38b98"
Nov 03 00:12:59.676032 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.676007    4294 scope.go:122] "RemoveContainer" containerID="02a9e8323a63b9d2981e06b1c6a74041cb12c22839f9eb18785395a91bb38b98"
Nov 03 00:12:59.676358 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:12:59.676339    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02a9e8323a63b9d2981e06b1c6a74041cb12c22839f9eb18785395a91bb38b98\": container with ID starting with 02a9e8323a63b9d2981e06b1c6a74041cb12c22839f9eb18785395a91bb38b98 not found: ID does not exist" containerID="02a9e8323a63b9d2981e06b1c6a74041cb12c22839f9eb18785395a91bb38b98"
Nov 03 00:12:59.676426 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.676372    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02a9e8323a63b9d2981e06b1c6a74041cb12c22839f9eb18785395a91bb38b98"} err="failed to get container status \"02a9e8323a63b9d2981e06b1c6a74041cb12c22839f9eb18785395a91bb38b98\": rpc error: code = NotFound desc = could not find container \"02a9e8323a63b9d2981e06b1c6a74041cb12c22839f9eb18785395a91bb38b98\": container with ID starting with 02a9e8323a63b9d2981e06b1c6a74041cb12c22839f9eb18785395a91bb38b98 not found: ID does not exist"
Nov 03 00:12:59.822104 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.822025    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"]
Nov 03 00:12:59.825128 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:12:59.825102    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-59f97a-openshift-default-5c887b5447-8jmsg"]
Nov 03 00:13:00.548217 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:00.548186    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b007881f-8d1a-487c-ab6f-0ad740dd047a" path="/var/lib/kubelet/pods/b007881f-8d1a-487c-ab6f-0ad740dd047a/volumes"
Nov 03 00:13:03.511120 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:03.511049    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:07.970591 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:07.970552    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:11.753003 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:13:11.752932    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:13:11.795116 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:11.795062    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:12.534284 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:12.534247    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:17.215652 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:17.215607    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:17.450045 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:17.450006    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:21.738122 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:21.738076    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:23.016925 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:23.016891    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:23.178822 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:23.178790    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:26.259073 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:26.259039    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:30.822246 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:30.822214    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:31.309774 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:31.309742    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:35.354641 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:35.354603    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:39.840713 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:39.840676    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:41.750561 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:13:41.750519    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:13:44.370474 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:44.370442    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:47.700104 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:47.700072    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:48.861688 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:48.861647    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:49.024457 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:49.024417    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:53.345722 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:53.345689    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:57.816582 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:13:57.816544    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:02.468149 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:02.468116    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:06.904931 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:06.904898    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:11.504226 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:11.504191    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:11.663507 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:11.663473    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:11.750461 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:14:11.750425    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:14:15.834024 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:15.833990    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"]
Nov 03 00:14:15.834595 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:15.834191    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj" podUID="9c23a4b6-9cde-465d-9539-41c313ccc802" containerName="istio-proxy" containerID="cri-o://e98478fade349d6cc1f39660277781690dbc1193c80a7967e6a3a0a9d6d0353a" gracePeriod=30
Nov 03 00:14:15.955853 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:15.955818    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:17.049210 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:17.049170    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:17.217752 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:17.217717    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:17.937097 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:17.937066    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:18.986955 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:18.986924    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"]
Nov 03 00:14:18.987567 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:18.987551    4294 cpu_manager.go:414] "containerMap: removing container" podUID="b007881f-8d1a-487c-ab6f-0ad740dd047a" containerName="istio-proxy"
Nov 03 00:14:18.987618 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:18.987567    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b007881f-8d1a-487c-ab6f-0ad740dd047a" containerName="istio-proxy"
Nov 03 00:14:18.987684 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:18.987675    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="b007881f-8d1a-487c-ab6f-0ad740dd047a" containerName="istio-proxy"
Nov 03 00:14:18.991020 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:18.991004    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:19.046207 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.046172    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"]
Nov 03 00:14:19.087937 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.087904    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-a51378\"/\"default-dockercfg-jxwpt\""
Nov 03 00:14:19.087937 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.087912    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-a51378\"/\"kube-root-ca.crt\""
Nov 03 00:14:19.088848 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.088829    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-a51378\"/\"openshift-service-ca.crt\""
Nov 03 00:14:19.089012 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.088992    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-a51378\"/\"sub-test-model-a51378-kserve-self-signed-certs\""
Nov 03 00:14:19.107602 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.107576    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:19.107602 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.107585    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a976b1a0-de58-4e35-a698-aadd48bd47cc-tls-certs\") pod \"sub-test-model-a51378-kserve-58d599dbc-t9wdn\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") " pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:19.107773 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.107616    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-model-cache\") pod \"sub-test-model-a51378-kserve-58d599dbc-t9wdn\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") " pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:19.107773 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.107700    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lht8m\" (UniqueName: \"kubernetes.io/projected/a976b1a0-de58-4e35-a698-aadd48bd47cc-kube-api-access-lht8m\") pod \"sub-test-model-a51378-kserve-58d599dbc-t9wdn\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") " pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:19.107773 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.107731    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-dshm\") pod \"sub-test-model-a51378-kserve-58d599dbc-t9wdn\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") " pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:19.107955 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.107879    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-tmp-dir\") pod \"sub-test-model-a51378-kserve-58d599dbc-t9wdn\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") " pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:19.107955 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.107918    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-home\") pod \"sub-test-model-a51378-kserve-58d599dbc-t9wdn\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") " pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:19.208699 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.208666    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-tmp-dir\") pod \"sub-test-model-a51378-kserve-58d599dbc-t9wdn\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") " pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:19.208699 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.208700    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-home\") pod \"sub-test-model-a51378-kserve-58d599dbc-t9wdn\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") " pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:19.208931 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.208784    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a976b1a0-de58-4e35-a698-aadd48bd47cc-tls-certs\") pod \"sub-test-model-a51378-kserve-58d599dbc-t9wdn\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") " pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:19.208931 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.208803    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-model-cache\") pod \"sub-test-model-a51378-kserve-58d599dbc-t9wdn\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") " pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:19.208931 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.208840    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lht8m\" (UniqueName: \"kubernetes.io/projected/a976b1a0-de58-4e35-a698-aadd48bd47cc-kube-api-access-lht8m\") pod \"sub-test-model-a51378-kserve-58d599dbc-t9wdn\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") " pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:19.208931 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.208873    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-dshm\") pod \"sub-test-model-a51378-kserve-58d599dbc-t9wdn\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") " pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:19.209224 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.209161    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-tmp-dir\") pod \"sub-test-model-a51378-kserve-58d599dbc-t9wdn\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") " pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:19.209224 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.209168    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-home\") pod \"sub-test-model-a51378-kserve-58d599dbc-t9wdn\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") " pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:19.209375 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.209229    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-model-cache\") pod \"sub-test-model-a51378-kserve-58d599dbc-t9wdn\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") " pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:19.211037 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.211017    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-dshm\") pod \"sub-test-model-a51378-kserve-58d599dbc-t9wdn\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") " pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:19.211359 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.211342    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a976b1a0-de58-4e35-a698-aadd48bd47cc-tls-certs\") pod \"sub-test-model-a51378-kserve-58d599dbc-t9wdn\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") " pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:19.263549 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.263531    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lht8m\" (UniqueName: \"kubernetes.io/projected/a976b1a0-de58-4e35-a698-aadd48bd47cc-kube-api-access-lht8m\") pod \"sub-test-model-a51378-kserve-58d599dbc-t9wdn\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") " pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:19.303111 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.303087    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:19.555149 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.555074    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:14:19.555299 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.555139    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj" podUID="9c23a4b6-9cde-465d-9539-41c313ccc802" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:14:19.671411 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:14:19.671382    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda976b1a0_de58_4e35_a698_aadd48bd47cc.slice/crio-b98b825d317c7747c68be03895214343c93d193aa951e6ce012f17803b519da4 WatchSource:0}: Error finding container b98b825d317c7747c68be03895214343c93d193aa951e6ce012f17803b519da4: Status 404 returned error can't find the container with id b98b825d317c7747c68be03895214343c93d193aa951e6ce012f17803b519da4
Nov 03 00:14:19.671910 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:19.671887    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"]
Nov 03 00:14:20.192820 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:20.192786    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn" event={"ID":"a976b1a0-de58-4e35-a698-aadd48bd47cc","Type":"ContainerStarted","Data":"82cf9c3d854ce4022bf241c6ced896315b34456c7b13dff4930bd684c5c81ccb"}
Nov 03 00:14:20.193227 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:20.192833    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn" event={"ID":"a976b1a0-de58-4e35-a698-aadd48bd47cc","Type":"ContainerStarted","Data":"b98b825d317c7747c68be03895214343c93d193aa951e6ce012f17803b519da4"}
Nov 03 00:14:20.303926 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:20.303878    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn" podStartSLOduration=2.303864602 podStartE2EDuration="2.303864602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:20.30101946 +0000 UTC m=+4899.869374248" watchObservedRunningTime="2026-11-03 00:14:20.303864602 +0000 UTC m=+4899.872219390"
Nov 03 00:14:20.304110 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:20.304061    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:20.789438 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:20.789407    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:21.105277 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.105254    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:14:21.201736 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.201708    4294 generic.go:356] "Generic (PLEG): container finished" podID="9c23a4b6-9cde-465d-9539-41c313ccc802" containerID="e98478fade349d6cc1f39660277781690dbc1193c80a7967e6a3a0a9d6d0353a" exitCode=0
Nov 03 00:14:21.202180 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.201746    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"
Nov 03 00:14:21.202180 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.201788    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj" event={"ID":"9c23a4b6-9cde-465d-9539-41c313ccc802","Type":"ContainerDied","Data":"e98478fade349d6cc1f39660277781690dbc1193c80a7967e6a3a0a9d6d0353a"}
Nov 03 00:14:21.202180 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.201850    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj" event={"ID":"9c23a4b6-9cde-465d-9539-41c313ccc802","Type":"ContainerDied","Data":"4bdb0119bded295c310158ca4ee2f23e29737a218f970f512e93b7b02add315b"}
Nov 03 00:14:21.202180 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.201884    4294 scope.go:122] "RemoveContainer" containerID="e98478fade349d6cc1f39660277781690dbc1193c80a7967e6a3a0a9d6d0353a"
Nov 03 00:14:21.211926 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.211907    4294 scope.go:122] "RemoveContainer" containerID="e98478fade349d6cc1f39660277781690dbc1193c80a7967e6a3a0a9d6d0353a"
Nov 03 00:14:21.212227 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:14:21.212208    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e98478fade349d6cc1f39660277781690dbc1193c80a7967e6a3a0a9d6d0353a\": container with ID starting with e98478fade349d6cc1f39660277781690dbc1193c80a7967e6a3a0a9d6d0353a not found: ID does not exist" containerID="e98478fade349d6cc1f39660277781690dbc1193c80a7967e6a3a0a9d6d0353a"
Nov 03 00:14:21.212312 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.212234    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e98478fade349d6cc1f39660277781690dbc1193c80a7967e6a3a0a9d6d0353a"} err="failed to get container status \"e98478fade349d6cc1f39660277781690dbc1193c80a7967e6a3a0a9d6d0353a\": rpc error: code = NotFound desc = could not find container \"e98478fade349d6cc1f39660277781690dbc1193c80a7967e6a3a0a9d6d0353a\": container with ID starting with e98478fade349d6cc1f39660277781690dbc1193c80a7967e6a3a0a9d6d0353a not found: ID does not exist"
Nov 03 00:14:21.231590 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.231564    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-token\" (UniqueName: \"kubernetes.io/projected/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-token\") pod \"9c23a4b6-9cde-465d-9539-41c313ccc802\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") "
Nov 03 00:14:21.231807 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.231612    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-workload-certs\") pod \"9c23a4b6-9cde-465d-9539-41c313ccc802\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") "
Nov 03 00:14:21.231807 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.231675    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-workload-socket\") pod \"9c23a4b6-9cde-465d-9539-41c313ccc802\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") "
Nov 03 00:14:21.231807 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.231757    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c23a4b6-9cde-465d-9539-41c313ccc802-kube-api-access-gx9fj\" (UniqueName: \"kubernetes.io/projected/9c23a4b6-9cde-465d-9539-41c313ccc802-kube-api-access-gx9fj\") pod \"9c23a4b6-9cde-465d-9539-41c313ccc802\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") "
Nov 03 00:14:21.232031 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.231844    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-podinfo\") pod \"9c23a4b6-9cde-465d-9539-41c313ccc802\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") "
Nov 03 00:14:21.232031 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.231883    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c23a4b6-9cde-465d-9539-41c313ccc802-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c23a4b6-9cde-465d-9539-41c313ccc802-istiod-ca-cert\") pod \"9c23a4b6-9cde-465d-9539-41c313ccc802\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") "
Nov 03 00:14:21.232031 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.232013    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-data\") pod \"9c23a4b6-9cde-465d-9539-41c313ccc802\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") "
Nov 03 00:14:21.232169 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.232090    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-credential-socket\") pod \"9c23a4b6-9cde-465d-9539-41c313ccc802\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") "
Nov 03 00:14:21.232169 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.232127    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-envoy\") pod \"9c23a4b6-9cde-465d-9539-41c313ccc802\" (UID: \"9c23a4b6-9cde-465d-9539-41c313ccc802\") "
Nov 03 00:14:21.234215 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.232009    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-workload-certs" pod "9c23a4b6-9cde-465d-9539-41c313ccc802" (UID: "9c23a4b6-9cde-465d-9539-41c313ccc802"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:14:21.234215 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.232337    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c23a4b6-9cde-465d-9539-41c313ccc802-istiod-ca-cert" pod "9c23a4b6-9cde-465d-9539-41c313ccc802" (UID: "9c23a4b6-9cde-465d-9539-41c313ccc802"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:14:21.234404 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.233905    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-token" pod "9c23a4b6-9cde-465d-9539-41c313ccc802" (UID: "9c23a4b6-9cde-465d-9539-41c313ccc802"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:14:21.234404 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.234178    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-workload-socket" pod "9c23a4b6-9cde-465d-9539-41c313ccc802" (UID: "9c23a4b6-9cde-465d-9539-41c313ccc802"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:14:21.234404 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.234338    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c23a4b6-9cde-465d-9539-41c313ccc802-kube-api-access-gx9fj" pod "9c23a4b6-9cde-465d-9539-41c313ccc802" (UID: "9c23a4b6-9cde-465d-9539-41c313ccc802"). InnerVolumeSpecName "kube-api-access-gx9fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:14:21.234580 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.234420    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-data" pod "9c23a4b6-9cde-465d-9539-41c313ccc802" (UID: "9c23a4b6-9cde-465d-9539-41c313ccc802"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:14:21.234580 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.234467    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-podinfo" pod "9c23a4b6-9cde-465d-9539-41c313ccc802" (UID: "9c23a4b6-9cde-465d-9539-41c313ccc802"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:14:21.234580 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.234534    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-credential-socket" pod "9c23a4b6-9cde-465d-9539-41c313ccc802" (UID: "9c23a4b6-9cde-465d-9539-41c313ccc802"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:14:21.235266 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.235242    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-envoy" pod "9c23a4b6-9cde-465d-9539-41c313ccc802" (UID: "9c23a4b6-9cde-465d-9539-41c313ccc802"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:14:21.334467 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.334411    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:21.334467 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.334434    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c23a4b6-9cde-465d-9539-41c313ccc802-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:21.334467 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.334444    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:21.334467 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.334455    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:21.334467 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.334464    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:21.334717 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.334473    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c23a4b6-9cde-465d-9539-41c313ccc802-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:21.334717 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.334483    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:21.334717 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.334492    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c23a4b6-9cde-465d-9539-41c313ccc802-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:21.334717 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.334501    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx9fj\" (UniqueName: \"kubernetes.io/projected/9c23a4b6-9cde-465d-9539-41c313ccc802-kube-api-access-gx9fj\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:21.625655 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.625578    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"]
Nov 03 00:14:21.634338 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:21.634308    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-xgcsj"]
Nov 03 00:14:22.549433 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:22.549403    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c23a4b6-9cde-465d-9539-41c313ccc802" path="/var/lib/kubelet/pods/9c23a4b6-9cde-465d-9539-41c313ccc802/volumes"
Nov 03 00:14:23.601153 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:23.601099    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:25.257164 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:25.257135    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:29.303637 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:29.303605    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:29.316005 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:29.315967    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:30.170998 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:30.170946    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:30.265588 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:30.265560    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:14:30.953446 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:30.953412    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"]
Nov 03 00:14:30.953918 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:30.953641    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5" podUID="df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" containerName="istio-proxy" containerID="cri-o://ab1be8120cb8c17883d692bc2c9f23c9f860b9d6cab443a5d0cae981427995f8" gracePeriod=30
Nov 03 00:14:31.001192 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:31.001161    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"]
Nov 03 00:14:31.001391 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:31.001355    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl" podUID="1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" containerName="istio-proxy" containerID="cri-o://9c0fd3178fe34d6578050fcd34a0e3df3be7be9504e330de18390358fc03f96e" gracePeriod=30
Nov 03 00:14:33.526505 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:33.526464    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:14:33.526891 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:33.526523    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5" podUID="df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:14:34.663934 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:34.663902    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-cvprl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:14:34.664355 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:34.663952    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl" podUID="1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:14:35.034129 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:35.034094    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:36.284575 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.284552    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 03 00:14:36.287796 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.287778    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:14:36.291469 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.291447    4294 generic.go:356] "Generic (PLEG): container finished" podID="df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" containerID="ab1be8120cb8c17883d692bc2c9f23c9f860b9d6cab443a5d0cae981427995f8" exitCode=0
Nov 03 00:14:36.291595 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.291480    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"
Nov 03 00:14:36.291595 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.291532    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5" event={"ID":"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c","Type":"ContainerDied","Data":"ab1be8120cb8c17883d692bc2c9f23c9f860b9d6cab443a5d0cae981427995f8"}
Nov 03 00:14:36.291595 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.291591    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5" event={"ID":"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c","Type":"ContainerDied","Data":"7549354e367b7a630181c0dd68d933ea9f9a3238b476c11d12b6b83b2e6e9525"}
Nov 03 00:14:36.291772 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.291621    4294 scope.go:122] "RemoveContainer" containerID="ab1be8120cb8c17883d692bc2c9f23c9f860b9d6cab443a5d0cae981427995f8"
Nov 03 00:14:36.292696 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.292673    4294 generic.go:356] "Generic (PLEG): container finished" podID="1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" containerID="9c0fd3178fe34d6578050fcd34a0e3df3be7be9504e330de18390358fc03f96e" exitCode=0
Nov 03 00:14:36.292819 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.292708    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"
Nov 03 00:14:36.292819 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.292761    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl" event={"ID":"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2","Type":"ContainerDied","Data":"9c0fd3178fe34d6578050fcd34a0e3df3be7be9504e330de18390358fc03f96e"}
Nov 03 00:14:36.292819 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.292801    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl" event={"ID":"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2","Type":"ContainerDied","Data":"26ea640f77a5687010755686c2d44da587e43929c21292221f2dbccd999133a3"}
Nov 03 00:14:36.301296 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.301275    4294 scope.go:122] "RemoveContainer" containerID="ab1be8120cb8c17883d692bc2c9f23c9f860b9d6cab443a5d0cae981427995f8"
Nov 03 00:14:36.301597 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:14:36.301578    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab1be8120cb8c17883d692bc2c9f23c9f860b9d6cab443a5d0cae981427995f8\": container with ID starting with ab1be8120cb8c17883d692bc2c9f23c9f860b9d6cab443a5d0cae981427995f8 not found: ID does not exist" containerID="ab1be8120cb8c17883d692bc2c9f23c9f860b9d6cab443a5d0cae981427995f8"
Nov 03 00:14:36.301677 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.301604    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab1be8120cb8c17883d692bc2c9f23c9f860b9d6cab443a5d0cae981427995f8"} err="failed to get container status \"ab1be8120cb8c17883d692bc2c9f23c9f860b9d6cab443a5d0cae981427995f8\": rpc error: code = NotFound desc = could not find container \"ab1be8120cb8c17883d692bc2c9f23c9f860b9d6cab443a5d0cae981427995f8\": container with ID starting with ab1be8120cb8c17883d692bc2c9f23c9f860b9d6cab443a5d0cae981427995f8 not found: ID does not exist"
Nov 03 00:14:36.301677 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.301622    4294 scope.go:122] "RemoveContainer" containerID="9c0fd3178fe34d6578050fcd34a0e3df3be7be9504e330de18390358fc03f96e"
Nov 03 00:14:36.310373 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.310355    4294 scope.go:122] "RemoveContainer" containerID="9c0fd3178fe34d6578050fcd34a0e3df3be7be9504e330de18390358fc03f96e"
Nov 03 00:14:36.310589 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:14:36.310574    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c0fd3178fe34d6578050fcd34a0e3df3be7be9504e330de18390358fc03f96e\": container with ID starting with 9c0fd3178fe34d6578050fcd34a0e3df3be7be9504e330de18390358fc03f96e not found: ID does not exist" containerID="9c0fd3178fe34d6578050fcd34a0e3df3be7be9504e330de18390358fc03f96e"
Nov 03 00:14:36.310641 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.310594    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c0fd3178fe34d6578050fcd34a0e3df3be7be9504e330de18390358fc03f96e"} err="failed to get container status \"9c0fd3178fe34d6578050fcd34a0e3df3be7be9504e330de18390358fc03f96e\": rpc error: code = NotFound desc = could not find container \"9c0fd3178fe34d6578050fcd34a0e3df3be7be9504e330de18390358fc03f96e\": container with ID starting with 9c0fd3178fe34d6578050fcd34a0e3df3be7be9504e330de18390358fc03f96e not found: ID does not exist"
Nov 03 00:14:36.326526 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.326501    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:36.390670 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.390632    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-podinfo\") pod \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") "
Nov 03 00:14:36.390802 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.390715    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-envoy\") pod \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") "
Nov 03 00:14:36.390802 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.390748    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-kube-api-access-lh4rh\" (UniqueName: \"kubernetes.io/projected/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-kube-api-access-lh4rh\") pod \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") "
Nov 03 00:14:36.390802 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.390787    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-data\") pod \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") "
Nov 03 00:14:36.390927 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.390807    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istiod-ca-cert\") pod \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") "
Nov 03 00:14:36.390927 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.390851    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-podinfo\") pod \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") "
Nov 03 00:14:36.390927 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.390886    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-token\" (UniqueName: \"kubernetes.io/projected/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-token\") pod \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") "
Nov 03 00:14:36.390927 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.390909    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-workload-socket\") pod \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") "
Nov 03 00:14:36.391110 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.390948    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-envoy\") pod \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") "
Nov 03 00:14:36.391110 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.391031    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-credential-socket\") pod \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") "
Nov 03 00:14:36.391110 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.391077    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-workload-certs\") pod \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") "
Nov 03 00:14:36.391273 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.391119    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istiod-ca-cert\") pod \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") "
Nov 03 00:14:36.391273 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.391159    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-kube-api-access-4m6tb\" (UniqueName: \"kubernetes.io/projected/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-kube-api-access-4m6tb\") pod \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") "
Nov 03 00:14:36.391273 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.391209    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-workload-certs\") pod \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") "
Nov 03 00:14:36.391273 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.391253    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-credential-socket\") pod \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") "
Nov 03 00:14:36.391523 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.391299    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-token\" (UniqueName: \"kubernetes.io/projected/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-token\") pod \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") "
Nov 03 00:14:36.391523 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.391332    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-data\") pod \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\" (UID: \"df40aa5e-4808-47d6-a6c2-3d793e7c0f7c\") "
Nov 03 00:14:36.391523 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.391347    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istiod-ca-cert" pod "1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" (UID: "1f9dbf5e-0447-4975-b1f3-e74e374ee7a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:14:36.391523 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.391353    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-data" pod "1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" (UID: "1f9dbf5e-0447-4975-b1f3-e74e374ee7a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:14:36.391523 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.391362    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-workload-socket\") pod \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\" (UID: \"1f9dbf5e-0447-4975-b1f3-e74e374ee7a2\") "
Nov 03 00:14:36.391813 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.391587    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-workload-socket" pod "1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" (UID: "1f9dbf5e-0447-4975-b1f3-e74e374ee7a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:14:36.391813 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.391611    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-workload-certs" pod "1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" (UID: "1f9dbf5e-0447-4975-b1f3-e74e374ee7a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:14:36.392008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.391968    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-workload-certs" pod "df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" (UID: "df40aa5e-4808-47d6-a6c2-3d793e7c0f7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:14:36.392478 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.392450    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-data" pod "df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" (UID: "df40aa5e-4808-47d6-a6c2-3d793e7c0f7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:14:36.392640 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.392617    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-workload-socket" pod "df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" (UID: "df40aa5e-4808-47d6-a6c2-3d793e7c0f7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:14:36.392748 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.392703    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-credential-socket" pod "1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" (UID: "1f9dbf5e-0447-4975-b1f3-e74e374ee7a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:14:36.392748 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.392711    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istiod-ca-cert" pod "df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" (UID: "df40aa5e-4808-47d6-a6c2-3d793e7c0f7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:14:36.392955 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.392908    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-credential-socket" pod "df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" (UID: "df40aa5e-4808-47d6-a6c2-3d793e7c0f7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:14:36.393317 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.393289    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:36.393408 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.393324    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:36.393408 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.393340    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:36.393408 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.393354    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:36.393408 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.393368    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:36.393408 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.393382    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:36.393408 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.393395    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:36.393659 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.393409    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:36.393659 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.393423    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:36.393659 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.393437    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:36.393659 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.393519    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-podinfo" pod "df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" (UID: "df40aa5e-4808-47d6-a6c2-3d793e7c0f7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:14:36.395958 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.395795    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-envoy" pod "1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" (UID: "1f9dbf5e-0447-4975-b1f3-e74e374ee7a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:14:36.395958 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.395876    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-podinfo" pod "1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" (UID: "1f9dbf5e-0447-4975-b1f3-e74e374ee7a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:14:36.395958 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.395891    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-token" pod "1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" (UID: "1f9dbf5e-0447-4975-b1f3-e74e374ee7a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:14:36.395958 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.395933    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-token" pod "df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" (UID: "df40aa5e-4808-47d6-a6c2-3d793e7c0f7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:14:36.396181 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.396015    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-envoy" pod "df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" (UID: "df40aa5e-4808-47d6-a6c2-3d793e7c0f7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:14:36.396281 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.396257    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-kube-api-access-4m6tb" pod "1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" (UID: "1f9dbf5e-0447-4975-b1f3-e74e374ee7a2"). InnerVolumeSpecName "kube-api-access-4m6tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:14:36.396598 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.396574    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-kube-api-access-lh4rh" pod "df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" (UID: "df40aa5e-4808-47d6-a6c2-3d793e7c0f7c"). InnerVolumeSpecName "kube-api-access-lh4rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:14:36.494512 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.494487    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:36.494512 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.494512    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:36.494706 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.494523    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:36.494706 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.494532    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh4rh\" (UniqueName: \"kubernetes.io/projected/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-kube-api-access-lh4rh\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:36.494706 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.494542    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:36.494706 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.494551    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:36.494706 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.494561    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:36.494706 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.494570    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m6tb\" (UniqueName: \"kubernetes.io/projected/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2-kube-api-access-4m6tb\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:36.712881 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.712847    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"]
Nov 03 00:14:36.719191 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.719164    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-6xpg5"]
Nov 03 00:14:36.835841 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.835738    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"]
Nov 03 00:14:36.838205 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:36.838179    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-cvprl"]
Nov 03 00:14:38.547779 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:38.547745    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" path="/var/lib/kubelet/pods/1f9dbf5e-0447-4975-b1f3-e74e374ee7a2/volumes"
Nov 03 00:14:38.548302 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:38.548286    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" path="/var/lib/kubelet/pods/df40aa5e-4808-47d6-a6c2-3d793e7c0f7c/volumes"
Nov 03 00:14:38.853062 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:38.852969    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"]
Nov 03 00:14:38.853251 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:38.853228    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9" podUID="b7940146-b317-4fa3-9193-0aad4dd2a294" containerName="maas-api" containerID="cri-o://8444d3c980a64f78f3991a4646cdd9cc4dd93c082874a79f6f2aaec85431870c" gracePeriod=30
Nov 03 00:14:38.995412 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:38.995380    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:39.118238 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.118156    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"
Nov 03 00:14:39.227960 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.227910    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7940146-b317-4fa3-9193-0aad4dd2a294-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7940146-b317-4fa3-9193-0aad4dd2a294-metrics-certs\") pod \"b7940146-b317-4fa3-9193-0aad4dd2a294\" (UID: \"b7940146-b317-4fa3-9193-0aad4dd2a294\") "
Nov 03 00:14:39.228184 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.228003    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7940146-b317-4fa3-9193-0aad4dd2a294-kube-api-access-5bqbl\" (UniqueName: \"kubernetes.io/projected/b7940146-b317-4fa3-9193-0aad4dd2a294-kube-api-access-5bqbl\") pod \"b7940146-b317-4fa3-9193-0aad4dd2a294\" (UID: \"b7940146-b317-4fa3-9193-0aad4dd2a294\") "
Nov 03 00:14:39.228184 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.228031    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7940146-b317-4fa3-9193-0aad4dd2a294-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7940146-b317-4fa3-9193-0aad4dd2a294-maas-api-tls\") pod \"b7940146-b317-4fa3-9193-0aad4dd2a294\" (UID: \"b7940146-b317-4fa3-9193-0aad4dd2a294\") "
Nov 03 00:14:39.228184 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.228064    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7940146-b317-4fa3-9193-0aad4dd2a294-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7940146-b317-4fa3-9193-0aad4dd2a294-openshift-service-ca\") pod \"b7940146-b317-4fa3-9193-0aad4dd2a294\" (UID: \"b7940146-b317-4fa3-9193-0aad4dd2a294\") "
Nov 03 00:14:39.228553 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.228512    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7940146-b317-4fa3-9193-0aad4dd2a294-openshift-service-ca" pod "b7940146-b317-4fa3-9193-0aad4dd2a294" (UID: "b7940146-b317-4fa3-9193-0aad4dd2a294"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:14:39.229152 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.229131    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7940146-b317-4fa3-9193-0aad4dd2a294-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:39.231555 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.230943    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7940146-b317-4fa3-9193-0aad4dd2a294-maas-api-tls" pod "b7940146-b317-4fa3-9193-0aad4dd2a294" (UID: "b7940146-b317-4fa3-9193-0aad4dd2a294"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:14:39.231555 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.231503    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7940146-b317-4fa3-9193-0aad4dd2a294-metrics-certs" pod "b7940146-b317-4fa3-9193-0aad4dd2a294" (UID: "b7940146-b317-4fa3-9193-0aad4dd2a294"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:14:39.232926 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.232900    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7940146-b317-4fa3-9193-0aad4dd2a294-kube-api-access-5bqbl" pod "b7940146-b317-4fa3-9193-0aad4dd2a294" (UID: "b7940146-b317-4fa3-9193-0aad4dd2a294"). InnerVolumeSpecName "kube-api-access-5bqbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:14:39.316456 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.316420    4294 generic.go:356] "Generic (PLEG): container finished" podID="b7940146-b317-4fa3-9193-0aad4dd2a294" containerID="8444d3c980a64f78f3991a4646cdd9cc4dd93c082874a79f6f2aaec85431870c" exitCode=0
Nov 03 00:14:39.316607 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.316567    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9" event={"ID":"b7940146-b317-4fa3-9193-0aad4dd2a294","Type":"ContainerDied","Data":"8444d3c980a64f78f3991a4646cdd9cc4dd93c082874a79f6f2aaec85431870c"}
Nov 03 00:14:39.316607 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.316604    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9" event={"ID":"b7940146-b317-4fa3-9193-0aad4dd2a294","Type":"ContainerDied","Data":"05707a48547a2c447f1056a9fa41f1dc61eb71af28b8e0b6914731519becade3"}
Nov 03 00:14:39.316694 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.316626    4294 scope.go:122] "RemoveContainer" containerID="8444d3c980a64f78f3991a4646cdd9cc4dd93c082874a79f6f2aaec85431870c"
Nov 03 00:14:39.316745 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.316734    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"
Nov 03 00:14:39.330203 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.330180    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7940146-b317-4fa3-9193-0aad4dd2a294-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:39.330354 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.330206    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bqbl\" (UniqueName: \"kubernetes.io/projected/b7940146-b317-4fa3-9193-0aad4dd2a294-kube-api-access-5bqbl\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:39.330354 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.330221    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7940146-b317-4fa3-9193-0aad4dd2a294-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:14:39.334624 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.334540    4294 scope.go:122] "RemoveContainer" containerID="8444d3c980a64f78f3991a4646cdd9cc4dd93c082874a79f6f2aaec85431870c"
Nov 03 00:14:39.334919 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:14:39.334869    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8444d3c980a64f78f3991a4646cdd9cc4dd93c082874a79f6f2aaec85431870c\": container with ID starting with 8444d3c980a64f78f3991a4646cdd9cc4dd93c082874a79f6f2aaec85431870c not found: ID does not exist" containerID="8444d3c980a64f78f3991a4646cdd9cc4dd93c082874a79f6f2aaec85431870c"
Nov 03 00:14:39.335040 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.334909    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8444d3c980a64f78f3991a4646cdd9cc4dd93c082874a79f6f2aaec85431870c"} err="failed to get container status \"8444d3c980a64f78f3991a4646cdd9cc4dd93c082874a79f6f2aaec85431870c\": rpc error: code = NotFound desc = could not find container \"8444d3c980a64f78f3991a4646cdd9cc4dd93c082874a79f6f2aaec85431870c\": container with ID starting with 8444d3c980a64f78f3991a4646cdd9cc4dd93c082874a79f6f2aaec85431870c not found: ID does not exist"
Nov 03 00:14:39.483051 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.482062    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"]
Nov 03 00:14:39.487101 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:39.487008    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9"]
Nov 03 00:14:40.547706 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:40.547676    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7940146-b317-4fa3-9193-0aad4dd2a294" path="/var/lib/kubelet/pods/b7940146-b317-4fa3-9193-0aad4dd2a294/volumes"
Nov 03 00:14:40.654034 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:40.654007    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:41.750465 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:14:41.750429    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:14:42.037757 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:42.037719    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f668443-6f4cb5d9fc-8nrt9" podUID="b7940146-b317-4fa3-9193-0aad4dd2a294" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.202:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:14:45.184256 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:45.184218    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:48.162088 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:48.162052    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:48.334604 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:48.334568    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:49.356131 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:49.356098    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:49.726902 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:49.726818    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:50.027215 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:50.027186    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:50.394326 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:50.394254    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:51.113094 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:51.113061    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:54.765421 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:54.765383    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:56.785341 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:56.785300    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:56.833611 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:56.833577    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg"]
Nov 03 00:14:57.103209 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:57.103121    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:57.259618 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:57.259562    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:57.726084 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:57.726047    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:59.479572 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:14:59.479539    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:00.173900 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.173862    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r"]
Nov 03 00:15:00.174483 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.174468    4294 cpu_manager.go:414] "containerMap: removing container" podUID="b7940146-b317-4fa3-9193-0aad4dd2a294" containerName="maas-api"
Nov 03 00:15:00.174543 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.174487    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7940146-b317-4fa3-9193-0aad4dd2a294" containerName="maas-api"
Nov 03 00:15:00.174543 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.174528    4294 cpu_manager.go:414] "containerMap: removing container" podUID="df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" containerName="istio-proxy"
Nov 03 00:15:00.174543 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.174533    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" containerName="istio-proxy"
Nov 03 00:15:00.174655 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.174545    4294 cpu_manager.go:414] "containerMap: removing container" podUID="1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" containerName="istio-proxy"
Nov 03 00:15:00.174655 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.174550    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" containerName="istio-proxy"
Nov 03 00:15:00.174655 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.174562    4294 cpu_manager.go:414] "containerMap: removing container" podUID="9c23a4b6-9cde-465d-9539-41c313ccc802" containerName="istio-proxy"
Nov 03 00:15:00.174655 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.174567    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c23a4b6-9cde-465d-9539-41c313ccc802" containerName="istio-proxy"
Nov 03 00:15:00.174655 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.174654    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="df40aa5e-4808-47d6-a6c2-3d793e7c0f7c" containerName="istio-proxy"
Nov 03 00:15:00.174839 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.174666    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9dbf5e-0447-4975-b1f3-e74e374ee7a2" containerName="istio-proxy"
Nov 03 00:15:00.174839 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.174674    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c23a4b6-9cde-465d-9539-41c313ccc802" containerName="istio-proxy"
Nov 03 00:15:00.174839 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.174683    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7940146-b317-4fa3-9193-0aad4dd2a294" containerName="maas-api"
Nov 03 00:15:00.178119 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.178103    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r"
Nov 03 00:15:00.183732 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.183709    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn"]
Nov 03 00:15:00.187832 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.187803    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn"
Nov 03 00:15:00.200644 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.200621    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4"]
Nov 03 00:15:00.204343 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.204328    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4"
Nov 03 00:15:00.228097 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.228077    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5gt8\""
Nov 03 00:15:00.275858 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.275815    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/150d536c-5dbc-4acf-826c-0f19f3c57320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn\" (UID: \"150d536c-5dbc-4acf-826c-0f19f3c57320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn"
Nov 03 00:15:00.275858 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.275852    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d4c3bf2-7d02-496e-ad2e-665266c9bc1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r\" (UID: \"2d4c3bf2-7d02-496e-ad2e-665266c9bc1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r"
Nov 03 00:15:00.276144 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.276021    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-954z2\" (UniqueName: \"kubernetes.io/projected/4df8f166-1d50-4ff1-9965-43d332f18d63-kube-api-access-954z2\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4\" (UID: \"4df8f166-1d50-4ff1-9965-43d332f18d63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4"
Nov 03 00:15:00.276194 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.276169    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzgnr\" (UniqueName: \"kubernetes.io/projected/150d536c-5dbc-4acf-826c-0f19f3c57320-kube-api-access-qzgnr\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn\" (UID: \"150d536c-5dbc-4acf-826c-0f19f3c57320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn"
Nov 03 00:15:00.276236 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.276209    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvgpk\" (UniqueName: \"kubernetes.io/projected/2d4c3bf2-7d02-496e-ad2e-665266c9bc1d-kube-api-access-xvgpk\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r\" (UID: \"2d4c3bf2-7d02-496e-ad2e-665266c9bc1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r"
Nov 03 00:15:00.276292 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.276235    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4df8f166-1d50-4ff1-9965-43d332f18d63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4\" (UID: \"4df8f166-1d50-4ff1-9965-43d332f18d63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4"
Nov 03 00:15:00.301040 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.301005    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn"]
Nov 03 00:15:00.304869 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.304837    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4"]
Nov 03 00:15:00.359649 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.359611    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r"]
Nov 03 00:15:00.377166 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.377134    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzgnr\" (UniqueName: \"kubernetes.io/projected/150d536c-5dbc-4acf-826c-0f19f3c57320-kube-api-access-qzgnr\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn\" (UID: \"150d536c-5dbc-4acf-826c-0f19f3c57320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn"
Nov 03 00:15:00.377350 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.377174    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvgpk\" (UniqueName: \"kubernetes.io/projected/2d4c3bf2-7d02-496e-ad2e-665266c9bc1d-kube-api-access-xvgpk\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r\" (UID: \"2d4c3bf2-7d02-496e-ad2e-665266c9bc1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r"
Nov 03 00:15:00.377350 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.377194    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4df8f166-1d50-4ff1-9965-43d332f18d63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4\" (UID: \"4df8f166-1d50-4ff1-9965-43d332f18d63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4"
Nov 03 00:15:00.377350 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.377276    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/150d536c-5dbc-4acf-826c-0f19f3c57320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn\" (UID: \"150d536c-5dbc-4acf-826c-0f19f3c57320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn"
Nov 03 00:15:00.377543 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.377399    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d4c3bf2-7d02-496e-ad2e-665266c9bc1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r\" (UID: \"2d4c3bf2-7d02-496e-ad2e-665266c9bc1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r"
Nov 03 00:15:00.377543 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.377485    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-954z2\" (UniqueName: \"kubernetes.io/projected/4df8f166-1d50-4ff1-9965-43d332f18d63-kube-api-access-954z2\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4\" (UID: \"4df8f166-1d50-4ff1-9965-43d332f18d63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4"
Nov 03 00:15:00.378124 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.378101    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/150d536c-5dbc-4acf-826c-0f19f3c57320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn\" (UID: \"150d536c-5dbc-4acf-826c-0f19f3c57320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn"
Nov 03 00:15:00.378240 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.378101    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4df8f166-1d50-4ff1-9965-43d332f18d63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4\" (UID: \"4df8f166-1d50-4ff1-9965-43d332f18d63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4"
Nov 03 00:15:00.378240 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.378202    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d4c3bf2-7d02-496e-ad2e-665266c9bc1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r\" (UID: \"2d4c3bf2-7d02-496e-ad2e-665266c9bc1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r"
Nov 03 00:15:00.432775 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.432705    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzgnr\" (UniqueName: \"kubernetes.io/projected/150d536c-5dbc-4acf-826c-0f19f3c57320-kube-api-access-qzgnr\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn\" (UID: \"150d536c-5dbc-4acf-826c-0f19f3c57320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn"
Nov 03 00:15:00.432892 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.432816    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-954z2\" (UniqueName: \"kubernetes.io/projected/4df8f166-1d50-4ff1-9965-43d332f18d63-kube-api-access-954z2\") pod \"maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4\" (UID: \"4df8f166-1d50-4ff1-9965-43d332f18d63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4"
Nov 03 00:15:00.433003 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.432965    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvgpk\" (UniqueName: \"kubernetes.io/projected/2d4c3bf2-7d02-496e-ad2e-665266c9bc1d-kube-api-access-xvgpk\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r\" (UID: \"2d4c3bf2-7d02-496e-ad2e-665266c9bc1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r"
Nov 03 00:15:00.491908 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.491875    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r"
Nov 03 00:15:00.501071 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.501051    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn"
Nov 03 00:15:00.517120 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.517093    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4"
Nov 03 00:15:00.882867 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.882847    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn"]
Nov 03 00:15:00.884015 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:15:00.883991    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150d536c_5dbc_4acf_826c_0f19f3c57320.slice/crio-e5ac07f4f3059baa11640f3a6b4322b51c58868d9b5a76045345dc8b15eed3d2 WatchSource:0}: Error finding container e5ac07f4f3059baa11640f3a6b4322b51c58868d9b5a76045345dc8b15eed3d2: Status 404 returned error can't find the container with id e5ac07f4f3059baa11640f3a6b4322b51c58868d9b5a76045345dc8b15eed3d2
Nov 03 00:15:00.930396 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:15:00.930365    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df8f166_1d50_4ff1_9965_43d332f18d63.slice/crio-655acce948b6ad6f59097ef7d2f671f8f3941aa4b5b3d3193bf35dc7172a511e WatchSource:0}: Error finding container 655acce948b6ad6f59097ef7d2f671f8f3941aa4b5b3d3193bf35dc7172a511e: Status 404 returned error can't find the container with id 655acce948b6ad6f59097ef7d2f671f8f3941aa4b5b3d3193bf35dc7172a511e
Nov 03 00:15:00.930706 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.930680    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4"]
Nov 03 00:15:00.931300 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:15:00.931280    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4c3bf2_7d02_496e_ad2e_665266c9bc1d.slice/crio-431f8a26f462d49b9f265c46bd5fd4de41d95a125d0c2468ad6cb6e339de0f56 WatchSource:0}: Error finding container 431f8a26f462d49b9f265c46bd5fd4de41d95a125d0c2468ad6cb6e339de0f56: Status 404 returned error can't find the container with id 431f8a26f462d49b9f265c46bd5fd4de41d95a125d0c2468ad6cb6e339de0f56
Nov 03 00:15:00.934489 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:00.934469    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r"]
Nov 03 00:15:01.450925 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:01.450829    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn" event={"ID":"150d536c-5dbc-4acf-826c-0f19f3c57320","Type":"ContainerStarted","Data":"082a681c5a99ffda31838157578c5ac3089cb2e4637236ff2b5688ba9d6b9a5d"}
Nov 03 00:15:01.450925 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:01.450894    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn" event={"ID":"150d536c-5dbc-4acf-826c-0f19f3c57320","Type":"ContainerStarted","Data":"e5ac07f4f3059baa11640f3a6b4322b51c58868d9b5a76045345dc8b15eed3d2"}
Nov 03 00:15:01.452519 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:01.452491    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r" event={"ID":"2d4c3bf2-7d02-496e-ad2e-665266c9bc1d","Type":"ContainerStarted","Data":"eb1b35067a114415052ad3fecc61d87cd19bf9dcacf9efab022941870dcd9d27"}
Nov 03 00:15:01.452660 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:01.452532    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r" event={"ID":"2d4c3bf2-7d02-496e-ad2e-665266c9bc1d","Type":"ContainerStarted","Data":"431f8a26f462d49b9f265c46bd5fd4de41d95a125d0c2468ad6cb6e339de0f56"}
Nov 03 00:15:01.454211 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:01.454187    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4" event={"ID":"4df8f166-1d50-4ff1-9965-43d332f18d63","Type":"ContainerStarted","Data":"26f5a9ad709ec286c9ef4819287b936f4ff4e72061c5bdb0dc8ed194e0a842dc"}
Nov 03 00:15:01.454332 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:01.454231    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4" event={"ID":"4df8f166-1d50-4ff1-9965-43d332f18d63","Type":"ContainerStarted","Data":"655acce948b6ad6f59097ef7d2f671f8f3941aa4b5b3d3193bf35dc7172a511e"}
Nov 03 00:15:01.561287 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:01.561231    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn" podStartSLOduration=1.561218424 podStartE2EDuration="1.561218424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:01.557965485 +0000 UTC m=+4941.126320305" watchObservedRunningTime="2026-11-03 00:15:01.561218424 +0000 UTC m=+4941.129573211"
Nov 03 00:15:01.667552 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:01.667494    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r" podStartSLOduration=1.6674765950000001 podStartE2EDuration="1.667476595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:01.665868205 +0000 UTC m=+4941.234222992" watchObservedRunningTime="2026-11-03 00:15:01.667476595 +0000 UTC m=+4941.235831391"
Nov 03 00:15:01.776858 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:01.776811    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4" podStartSLOduration=1.776795006 podStartE2EDuration="1.776795006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:01.773439346 +0000 UTC m=+4941.341794140" watchObservedRunningTime="2026-11-03 00:15:01.776795006 +0000 UTC m=+4941.345149793"
Nov 03 00:15:01.834315 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:01.834281    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg" podUID="6665ed83-8dd6-4762-ad16-0b80d2b731f4" containerName="payload-pre-processing" containerID="cri-o://f6c249e382e9710d5e33bf2cada1ecf8203db32fedc3253988e2172247575e0c" gracePeriod=25
Nov 03 00:15:02.461923 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:02.461835    4294 generic.go:356] "Generic (PLEG): container finished" podID="4df8f166-1d50-4ff1-9965-43d332f18d63" containerID="26f5a9ad709ec286c9ef4819287b936f4ff4e72061c5bdb0dc8ed194e0a842dc" exitCode=60
Nov 03 00:15:02.461923 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:02.461912    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4" event={"ID":"4df8f166-1d50-4ff1-9965-43d332f18d63","Type":"ContainerDied","Data":"26f5a9ad709ec286c9ef4819287b936f4ff4e72061c5bdb0dc8ed194e0a842dc"}
Nov 03 00:15:02.462247 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:02.462227    4294 scope.go:122] "RemoveContainer" containerID="26f5a9ad709ec286c9ef4819287b936f4ff4e72061c5bdb0dc8ed194e0a842dc"
Nov 03 00:15:02.463686 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:02.463663    4294 generic.go:356] "Generic (PLEG): container finished" podID="150d536c-5dbc-4acf-826c-0f19f3c57320" containerID="082a681c5a99ffda31838157578c5ac3089cb2e4637236ff2b5688ba9d6b9a5d" exitCode=60
Nov 03 00:15:02.463809 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:02.463689    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn" event={"ID":"150d536c-5dbc-4acf-826c-0f19f3c57320","Type":"ContainerDied","Data":"082a681c5a99ffda31838157578c5ac3089cb2e4637236ff2b5688ba9d6b9a5d"}
Nov 03 00:15:02.463969 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:02.463953    4294 scope.go:122] "RemoveContainer" containerID="082a681c5a99ffda31838157578c5ac3089cb2e4637236ff2b5688ba9d6b9a5d"
Nov 03 00:15:02.465373 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:02.465355    4294 generic.go:356] "Generic (PLEG): container finished" podID="2d4c3bf2-7d02-496e-ad2e-665266c9bc1d" containerID="eb1b35067a114415052ad3fecc61d87cd19bf9dcacf9efab022941870dcd9d27" exitCode=60
Nov 03 00:15:02.465451 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:02.465436    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r" event={"ID":"2d4c3bf2-7d02-496e-ad2e-665266c9bc1d","Type":"ContainerDied","Data":"eb1b35067a114415052ad3fecc61d87cd19bf9dcacf9efab022941870dcd9d27"}
Nov 03 00:15:02.465628 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:02.465599    4294 scope.go:122] "RemoveContainer" containerID="eb1b35067a114415052ad3fecc61d87cd19bf9dcacf9efab022941870dcd9d27"
Nov 03 00:15:02.922510 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:02.922474    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:03.473125 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:03.473094    4294 generic.go:356] "Generic (PLEG): container finished" podID="4df8f166-1d50-4ff1-9965-43d332f18d63" containerID="0c774eaab3971160c2718c4d58332cac91ca424d3380881159fb435a3a469669" exitCode=60
Nov 03 00:15:03.473286 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:03.473176    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4" event={"ID":"4df8f166-1d50-4ff1-9965-43d332f18d63","Type":"ContainerDied","Data":"0c774eaab3971160c2718c4d58332cac91ca424d3380881159fb435a3a469669"}
Nov 03 00:15:03.473286 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:03.473234    4294 scope.go:122] "RemoveContainer" containerID="26f5a9ad709ec286c9ef4819287b936f4ff4e72061c5bdb0dc8ed194e0a842dc"
Nov 03 00:15:03.473543 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:03.473527    4294 scope.go:122] "RemoveContainer" containerID="0c774eaab3971160c2718c4d58332cac91ca424d3380881159fb435a3a469669"
Nov 03 00:15:03.473900 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:15:03.473876    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4_odh-ai-gateway-infra(4df8f166-1d50-4ff1-9965-43d332f18d63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4" podUID="4df8f166-1d50-4ff1-9965-43d332f18d63"
Nov 03 00:15:03.475392 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:03.475372    4294 generic.go:356] "Generic (PLEG): container finished" podID="150d536c-5dbc-4acf-826c-0f19f3c57320" containerID="dd8efee6335de4927f5ab50382d587d73f6f4f537610000364e5b13cb9b33452" exitCode=60
Nov 03 00:15:03.475495 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:03.475462    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn" event={"ID":"150d536c-5dbc-4acf-826c-0f19f3c57320","Type":"ContainerDied","Data":"dd8efee6335de4927f5ab50382d587d73f6f4f537610000364e5b13cb9b33452"}
Nov 03 00:15:03.475798 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:03.475782    4294 scope.go:122] "RemoveContainer" containerID="dd8efee6335de4927f5ab50382d587d73f6f4f537610000364e5b13cb9b33452"
Nov 03 00:15:03.476083 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:15:03.476064    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn_odh-ai-gateway-infra(150d536c-5dbc-4acf-826c-0f19f3c57320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn" podUID="150d536c-5dbc-4acf-826c-0f19f3c57320"
Nov 03 00:15:03.477224 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:03.477206    4294 generic.go:356] "Generic (PLEG): container finished" podID="2d4c3bf2-7d02-496e-ad2e-665266c9bc1d" containerID="e35f08842e4259f3897cbd1bee499a3065834b85d808441fe1ff7351a1f8658b" exitCode=60
Nov 03 00:15:03.477354 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:03.477233    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r" event={"ID":"2d4c3bf2-7d02-496e-ad2e-665266c9bc1d","Type":"ContainerDied","Data":"e35f08842e4259f3897cbd1bee499a3065834b85d808441fe1ff7351a1f8658b"}
Nov 03 00:15:03.477508 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:03.477495    4294 scope.go:122] "RemoveContainer" containerID="e35f08842e4259f3897cbd1bee499a3065834b85d808441fe1ff7351a1f8658b"
Nov 03 00:15:03.477728 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:15:03.477710    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r_odh-ai-gateway-infra(2d4c3bf2-7d02-496e-ad2e-665266c9bc1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r" podUID="2d4c3bf2-7d02-496e-ad2e-665266c9bc1d"
Nov 03 00:15:03.568649 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:03.568626    4294 scope.go:122] "RemoveContainer" containerID="082a681c5a99ffda31838157578c5ac3089cb2e4637236ff2b5688ba9d6b9a5d"
Nov 03 00:15:03.582519 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:03.582492    4294 scope.go:122] "RemoveContainer" containerID="eb1b35067a114415052ad3fecc61d87cd19bf9dcacf9efab022941870dcd9d27"
Nov 03 00:15:04.151528 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:04.151494    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:04.485565 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:04.485476    4294 scope.go:122] "RemoveContainer" containerID="0c774eaab3971160c2718c4d58332cac91ca424d3380881159fb435a3a469669"
Nov 03 00:15:04.485848 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:15:04.485818    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4_odh-ai-gateway-infra(4df8f166-1d50-4ff1-9965-43d332f18d63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4" podUID="4df8f166-1d50-4ff1-9965-43d332f18d63"
Nov 03 00:15:04.487082 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:04.487056    4294 scope.go:122] "RemoveContainer" containerID="dd8efee6335de4927f5ab50382d587d73f6f4f537610000364e5b13cb9b33452"
Nov 03 00:15:04.487343 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:15:04.487319    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn_odh-ai-gateway-infra(150d536c-5dbc-4acf-826c-0f19f3c57320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn" podUID="150d536c-5dbc-4acf-826c-0f19f3c57320"
Nov 03 00:15:04.488493 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:04.488475    4294 scope.go:122] "RemoveContainer" containerID="e35f08842e4259f3897cbd1bee499a3065834b85d808441fe1ff7351a1f8658b"
Nov 03 00:15:04.488684 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:15:04.488669    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r_odh-ai-gateway-infra(2d4c3bf2-7d02-496e-ad2e-665266c9bc1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r" podUID="2d4c3bf2-7d02-496e-ad2e-665266c9bc1d"
Nov 03 00:15:08.874019 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:08.872841    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:10.278445 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:10.278404    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg" podUID="6665ed83-8dd6-4762-ad16-0b80d2b731f4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:15:11.763688 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:15:11.760764    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:15:13.565087 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:13.565055    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:13.724391 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:13.724355    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:14.541150 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:14.541119    4294 scope.go:122] "RemoveContainer" containerID="0c774eaab3971160c2718c4d58332cac91ca424d3380881159fb435a3a469669"
Nov 03 00:15:15.560794 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:15.560763    4294 generic.go:356] "Generic (PLEG): container finished" podID="4df8f166-1d50-4ff1-9965-43d332f18d63" containerID="59fa1e348294a7cf12b121875bf19d4775ab782ca74019c4520a21098bb4a250" exitCode=60
Nov 03 00:15:15.561221 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:15.560842    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4" event={"ID":"4df8f166-1d50-4ff1-9965-43d332f18d63","Type":"ContainerDied","Data":"59fa1e348294a7cf12b121875bf19d4775ab782ca74019c4520a21098bb4a250"}
Nov 03 00:15:15.561221 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:15.560906    4294 scope.go:122] "RemoveContainer" containerID="0c774eaab3971160c2718c4d58332cac91ca424d3380881159fb435a3a469669"
Nov 03 00:15:15.561368 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:15.561353    4294 scope.go:122] "RemoveContainer" containerID="59fa1e348294a7cf12b121875bf19d4775ab782ca74019c4520a21098bb4a250"
Nov 03 00:15:15.561688 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:15:15.561663    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4_odh-ai-gateway-infra(4df8f166-1d50-4ff1-9965-43d332f18d63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4" podUID="4df8f166-1d50-4ff1-9965-43d332f18d63"
Nov 03 00:15:16.541381 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:16.541350    4294 scope.go:122] "RemoveContainer" containerID="dd8efee6335de4927f5ab50382d587d73f6f4f537610000364e5b13cb9b33452"
Nov 03 00:15:16.678744 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:16.678714    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4"]
Nov 03 00:15:16.822570 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:16.822546    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4"
Nov 03 00:15:16.894393 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:16.894369    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df8f166-1d50-4ff1-9965-43d332f18d63-kube-api-access-954z2\" (UniqueName: \"kubernetes.io/projected/4df8f166-1d50-4ff1-9965-43d332f18d63-kube-api-access-954z2\") pod \"4df8f166-1d50-4ff1-9965-43d332f18d63\" (UID: \"4df8f166-1d50-4ff1-9965-43d332f18d63\") "
Nov 03 00:15:16.894546 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:16.894412    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4df8f166-1d50-4ff1-9965-43d332f18d63-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4df8f166-1d50-4ff1-9965-43d332f18d63-maas-api-service-ca\") pod \"4df8f166-1d50-4ff1-9965-43d332f18d63\" (UID: \"4df8f166-1d50-4ff1-9965-43d332f18d63\") "
Nov 03 00:15:16.894767 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:16.894746    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df8f166-1d50-4ff1-9965-43d332f18d63-maas-api-service-ca" pod "4df8f166-1d50-4ff1-9965-43d332f18d63" (UID: "4df8f166-1d50-4ff1-9965-43d332f18d63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:16.894918 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:16.894903    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4df8f166-1d50-4ff1-9965-43d332f18d63-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:15:16.896297 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:16.896279    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df8f166-1d50-4ff1-9965-43d332f18d63-kube-api-access-954z2" pod "4df8f166-1d50-4ff1-9965-43d332f18d63" (UID: "4df8f166-1d50-4ff1-9965-43d332f18d63"). InnerVolumeSpecName "kube-api-access-954z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:16.995923 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:16.995889    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-954z2\" (UniqueName: \"kubernetes.io/projected/4df8f166-1d50-4ff1-9965-43d332f18d63-kube-api-access-954z2\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:15:17.032004 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.031944    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"]
Nov 03 00:15:17.032217 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.032194    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj" podUID="4d799ed5-6504-4440-aa0f-84ba25128c25" containerName="maas-api" containerID="cri-o://a4e1499cf29d501f21a352d6ff10f424df259be8373896e42398e705f2840fd9" gracePeriod=30
Nov 03 00:15:17.116091 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:15:17.116060    4294 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d799ed5_6504_4440_aa0f_84ba25128c25.slice/crio-conmon-a4e1499cf29d501f21a352d6ff10f424df259be8373896e42398e705f2840fd9.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 00:15:17.161465 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.161436    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:17.201003 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.200961    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn"]
Nov 03 00:15:17.263077 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.263053    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"
Nov 03 00:15:17.302898 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.302824    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d799ed5-6504-4440-aa0f-84ba25128c25-kube-api-access-rqvqg\" (UniqueName: \"kubernetes.io/projected/4d799ed5-6504-4440-aa0f-84ba25128c25-kube-api-access-rqvqg\") pod \"4d799ed5-6504-4440-aa0f-84ba25128c25\" (UID: \"4d799ed5-6504-4440-aa0f-84ba25128c25\") "
Nov 03 00:15:17.302898 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.302883    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d799ed5-6504-4440-aa0f-84ba25128c25-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d799ed5-6504-4440-aa0f-84ba25128c25-metrics-certs\") pod \"4d799ed5-6504-4440-aa0f-84ba25128c25\" (UID: \"4d799ed5-6504-4440-aa0f-84ba25128c25\") "
Nov 03 00:15:17.303143 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.302959    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d799ed5-6504-4440-aa0f-84ba25128c25-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d799ed5-6504-4440-aa0f-84ba25128c25-maas-api-tls\") pod \"4d799ed5-6504-4440-aa0f-84ba25128c25\" (UID: \"4d799ed5-6504-4440-aa0f-84ba25128c25\") "
Nov 03 00:15:17.303143 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.303026    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d799ed5-6504-4440-aa0f-84ba25128c25-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d799ed5-6504-4440-aa0f-84ba25128c25-openshift-service-ca\") pod \"4d799ed5-6504-4440-aa0f-84ba25128c25\" (UID: \"4d799ed5-6504-4440-aa0f-84ba25128c25\") "
Nov 03 00:15:17.303461 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.303434    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d799ed5-6504-4440-aa0f-84ba25128c25-openshift-service-ca" pod "4d799ed5-6504-4440-aa0f-84ba25128c25" (UID: "4d799ed5-6504-4440-aa0f-84ba25128c25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:17.304922 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.304903    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d799ed5-6504-4440-aa0f-84ba25128c25-metrics-certs" pod "4d799ed5-6504-4440-aa0f-84ba25128c25" (UID: "4d799ed5-6504-4440-aa0f-84ba25128c25"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:15:17.305035 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.304950    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d799ed5-6504-4440-aa0f-84ba25128c25-maas-api-tls" pod "4d799ed5-6504-4440-aa0f-84ba25128c25" (UID: "4d799ed5-6504-4440-aa0f-84ba25128c25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:15:17.305035 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.304992    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d799ed5-6504-4440-aa0f-84ba25128c25-kube-api-access-rqvqg" pod "4d799ed5-6504-4440-aa0f-84ba25128c25" (UID: "4d799ed5-6504-4440-aa0f-84ba25128c25"). InnerVolumeSpecName "kube-api-access-rqvqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:17.404210 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.404183    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d799ed5-6504-4440-aa0f-84ba25128c25-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:15:17.404327 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.404214    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d799ed5-6504-4440-aa0f-84ba25128c25-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:15:17.404327 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.404229    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqvqg\" (UniqueName: \"kubernetes.io/projected/4d799ed5-6504-4440-aa0f-84ba25128c25-kube-api-access-rqvqg\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:15:17.404327 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.404243    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d799ed5-6504-4440-aa0f-84ba25128c25-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:15:17.541428 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.541400    4294 scope.go:122] "RemoveContainer" containerID="e35f08842e4259f3897cbd1bee499a3065834b85d808441fe1ff7351a1f8658b"
Nov 03 00:15:17.578554 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.578519    4294 generic.go:356] "Generic (PLEG): container finished" podID="150d536c-5dbc-4acf-826c-0f19f3c57320" containerID="a902ba8d5369655c3d2eb52b932b98eed7121a76196fd39a3f6c4cb611e8654e" exitCode=6
Nov 03 00:15:17.578667 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.578602    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn" event={"ID":"150d536c-5dbc-4acf-826c-0f19f3c57320","Type":"ContainerDied","Data":"a902ba8d5369655c3d2eb52b932b98eed7121a76196fd39a3f6c4cb611e8654e"}
Nov 03 00:15:17.578745 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.578672    4294 scope.go:122] "RemoveContainer" containerID="dd8efee6335de4927f5ab50382d587d73f6f4f537610000364e5b13cb9b33452"
Nov 03 00:15:17.580574 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.580554    4294 generic.go:356] "Generic (PLEG): container finished" podID="4d799ed5-6504-4440-aa0f-84ba25128c25" containerID="a4e1499cf29d501f21a352d6ff10f424df259be8373896e42398e705f2840fd9" exitCode=0
Nov 03 00:15:17.580708 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.580593    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj" event={"ID":"4d799ed5-6504-4440-aa0f-84ba25128c25","Type":"ContainerDied","Data":"a4e1499cf29d501f21a352d6ff10f424df259be8373896e42398e705f2840fd9"}
Nov 03 00:15:17.580708 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.580624    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"
Nov 03 00:15:17.580708 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.580661    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj" event={"ID":"4d799ed5-6504-4440-aa0f-84ba25128c25","Type":"ContainerDied","Data":"09d96c871ff01d4182b09b4fe2f70f55e2bc5899c6ebc1787dc402852d21f1c4"}
Nov 03 00:15:17.582190 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.582168    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4" event={"ID":"4df8f166-1d50-4ff1-9965-43d332f18d63","Type":"ContainerDied","Data":"655acce948b6ad6f59097ef7d2f671f8f3941aa4b5b3d3193bf35dc7172a511e"}
Nov 03 00:15:17.582190 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.582186    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4"
Nov 03 00:15:17.601495 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.601462    4294 scope.go:122] "RemoveContainer" containerID="a4e1499cf29d501f21a352d6ff10f424df259be8373896e42398e705f2840fd9"
Nov 03 00:15:17.611533 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.610858    4294 scope.go:122] "RemoveContainer" containerID="a4e1499cf29d501f21a352d6ff10f424df259be8373896e42398e705f2840fd9"
Nov 03 00:15:17.611646 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:15:17.611603    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e1499cf29d501f21a352d6ff10f424df259be8373896e42398e705f2840fd9\": container with ID starting with a4e1499cf29d501f21a352d6ff10f424df259be8373896e42398e705f2840fd9 not found: ID does not exist" containerID="a4e1499cf29d501f21a352d6ff10f424df259be8373896e42398e705f2840fd9"
Nov 03 00:15:17.611723 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.611640    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e1499cf29d501f21a352d6ff10f424df259be8373896e42398e705f2840fd9"} err="failed to get container status \"a4e1499cf29d501f21a352d6ff10f424df259be8373896e42398e705f2840fd9\": rpc error: code = NotFound desc = could not find container \"a4e1499cf29d501f21a352d6ff10f424df259be8373896e42398e705f2840fd9\": container with ID starting with a4e1499cf29d501f21a352d6ff10f424df259be8373896e42398e705f2840fd9 not found: ID does not exist"
Nov 03 00:15:17.611723 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.611665    4294 scope.go:122] "RemoveContainer" containerID="59fa1e348294a7cf12b121875bf19d4775ab782ca74019c4520a21098bb4a250"
Nov 03 00:15:17.783074 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.783049    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn"
Nov 03 00:15:17.794788 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.794760    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"]
Nov 03 00:15:17.797750 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.797730    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-c5148c-5d785959b5-qm8fj"]
Nov 03 00:15:17.809269 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.809216    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/150d536c-5dbc-4acf-826c-0f19f3c57320-kube-api-access-qzgnr\" (UniqueName: \"kubernetes.io/projected/150d536c-5dbc-4acf-826c-0f19f3c57320-kube-api-access-qzgnr\") pod \"150d536c-5dbc-4acf-826c-0f19f3c57320\" (UID: \"150d536c-5dbc-4acf-826c-0f19f3c57320\") "
Nov 03 00:15:17.809404 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.809390    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/150d536c-5dbc-4acf-826c-0f19f3c57320-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/150d536c-5dbc-4acf-826c-0f19f3c57320-maas-api-service-ca\") pod \"150d536c-5dbc-4acf-826c-0f19f3c57320\" (UID: \"150d536c-5dbc-4acf-826c-0f19f3c57320\") "
Nov 03 00:15:17.809700 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.809678    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150d536c-5dbc-4acf-826c-0f19f3c57320-maas-api-service-ca" pod "150d536c-5dbc-4acf-826c-0f19f3c57320" (UID: "150d536c-5dbc-4acf-826c-0f19f3c57320"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:17.811383 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.811353    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150d536c-5dbc-4acf-826c-0f19f3c57320-kube-api-access-qzgnr" pod "150d536c-5dbc-4acf-826c-0f19f3c57320" (UID: "150d536c-5dbc-4acf-826c-0f19f3c57320"). InnerVolumeSpecName "kube-api-access-qzgnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:17.902244 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.902209    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4"]
Nov 03 00:15:17.904940 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.904915    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-14b321-29894415-8gls4"]
Nov 03 00:15:17.910705 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.910686    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/150d536c-5dbc-4acf-826c-0f19f3c57320-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:15:17.910786 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:17.910707    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzgnr\" (UniqueName: \"kubernetes.io/projected/150d536c-5dbc-4acf-826c-0f19f3c57320-kube-api-access-qzgnr\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:15:18.548375 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:18.548341    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d799ed5-6504-4440-aa0f-84ba25128c25" path="/var/lib/kubelet/pods/4d799ed5-6504-4440-aa0f-84ba25128c25/volumes"
Nov 03 00:15:18.548823 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:18.548803    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4df8f166-1d50-4ff1-9965-43d332f18d63" path="/var/lib/kubelet/pods/4df8f166-1d50-4ff1-9965-43d332f18d63/volumes"
Nov 03 00:15:18.589868 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:18.589843    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn"
Nov 03 00:15:18.589868 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:18.589857    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn" event={"ID":"150d536c-5dbc-4acf-826c-0f19f3c57320","Type":"ContainerDied","Data":"e5ac07f4f3059baa11640f3a6b4322b51c58868d9b5a76045345dc8b15eed3d2"}
Nov 03 00:15:18.590108 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:18.589922    4294 scope.go:122] "RemoveContainer" containerID="a902ba8d5369655c3d2eb52b932b98eed7121a76196fd39a3f6c4cb611e8654e"
Nov 03 00:15:18.592793 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:18.592764    4294 generic.go:356] "Generic (PLEG): container finished" podID="2d4c3bf2-7d02-496e-ad2e-665266c9bc1d" containerID="e961ca2e37fd5b090adb9dd3b2d6b91ff8edd4caac54599f0bb07b35b2cf9e47" exitCode=60
Nov 03 00:15:18.592906 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:18.592807    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r" event={"ID":"2d4c3bf2-7d02-496e-ad2e-665266c9bc1d","Type":"ContainerDied","Data":"e961ca2e37fd5b090adb9dd3b2d6b91ff8edd4caac54599f0bb07b35b2cf9e47"}
Nov 03 00:15:18.593505 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:18.593215    4294 scope.go:122] "RemoveContainer" containerID="e961ca2e37fd5b090adb9dd3b2d6b91ff8edd4caac54599f0bb07b35b2cf9e47"
Nov 03 00:15:18.593505 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:15:18.593471    4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r_odh-ai-gateway-infra(2d4c3bf2-7d02-496e-ad2e-665266c9bc1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r" podUID="2d4c3bf2-7d02-496e-ad2e-665266c9bc1d"
Nov 03 00:15:18.617740 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:18.617719    4294 scope.go:122] "RemoveContainer" containerID="e35f08842e4259f3897cbd1bee499a3065834b85d808441fe1ff7351a1f8658b"
Nov 03 00:15:18.748074 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:18.748044    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn"]
Nov 03 00:15:18.753614 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:18.753591    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c5148c-29894415-lb7gn"]
Nov 03 00:15:18.930177 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:18.930134    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:19.865639 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:19.865610    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r"]
Nov 03 00:15:20.016338 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:20.016312    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r"
Nov 03 00:15:20.036239 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:20.036215    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d4c3bf2-7d02-496e-ad2e-665266c9bc1d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d4c3bf2-7d02-496e-ad2e-665266c9bc1d-maas-api-service-ca\") pod \"2d4c3bf2-7d02-496e-ad2e-665266c9bc1d\" (UID: \"2d4c3bf2-7d02-496e-ad2e-665266c9bc1d\") "
Nov 03 00:15:20.036380 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:20.036266    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4c3bf2-7d02-496e-ad2e-665266c9bc1d-kube-api-access-xvgpk\" (UniqueName: \"kubernetes.io/projected/2d4c3bf2-7d02-496e-ad2e-665266c9bc1d-kube-api-access-xvgpk\") pod \"2d4c3bf2-7d02-496e-ad2e-665266c9bc1d\" (UID: \"2d4c3bf2-7d02-496e-ad2e-665266c9bc1d\") "
Nov 03 00:15:20.036565 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:20.036545    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d4c3bf2-7d02-496e-ad2e-665266c9bc1d-maas-api-service-ca" pod "2d4c3bf2-7d02-496e-ad2e-665266c9bc1d" (UID: "2d4c3bf2-7d02-496e-ad2e-665266c9bc1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:20.036785 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:20.036772    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d4c3bf2-7d02-496e-ad2e-665266c9bc1d-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:15:20.038311 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:20.038292    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4c3bf2-7d02-496e-ad2e-665266c9bc1d-kube-api-access-xvgpk" pod "2d4c3bf2-7d02-496e-ad2e-665266c9bc1d" (UID: "2d4c3bf2-7d02-496e-ad2e-665266c9bc1d"). InnerVolumeSpecName "kube-api-access-xvgpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:20.137968 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:20.137889    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvgpk\" (UniqueName: \"kubernetes.io/projected/2d4c3bf2-7d02-496e-ad2e-665266c9bc1d-kube-api-access-xvgpk\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:15:20.278511 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:20.278471    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg" podUID="6665ed83-8dd6-4762-ad16-0b80d2b731f4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:15:20.547836 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:20.547803    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="150d536c-5dbc-4acf-826c-0f19f3c57320" path="/var/lib/kubelet/pods/150d536c-5dbc-4acf-826c-0f19f3c57320/volumes"
Nov 03 00:15:20.617669 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:20.617641    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r"
Nov 03 00:15:20.617835 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:20.617686    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r" event={"ID":"2d4c3bf2-7d02-496e-ad2e-665266c9bc1d","Type":"ContainerDied","Data":"431f8a26f462d49b9f265c46bd5fd4de41d95a125d0c2468ad6cb6e339de0f56"}
Nov 03 00:15:20.617835 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:20.617765    4294 scope.go:122] "RemoveContainer" containerID="e961ca2e37fd5b090adb9dd3b2d6b91ff8edd4caac54599f0bb07b35b2cf9e47"
Nov 03 00:15:20.775480 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:20.775443    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r"]
Nov 03 00:15:20.781216 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:20.781167    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7ca60f-29894415-psx5r"]
Nov 03 00:15:22.092011 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:22.091943    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg"
Nov 03 00:15:22.158897 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:22.158868    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6665ed83-8dd6-4762-ad16-0b80d2b731f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6665ed83-8dd6-4762-ad16-0b80d2b731f4-plugins-config-volume\") pod \"6665ed83-8dd6-4762-ad16-0b80d2b731f4\" (UID: \"6665ed83-8dd6-4762-ad16-0b80d2b731f4\") "
Nov 03 00:15:22.159091 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:22.158924    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6665ed83-8dd6-4762-ad16-0b80d2b731f4-kube-api-access-bhgsf\" (UniqueName: \"kubernetes.io/projected/6665ed83-8dd6-4762-ad16-0b80d2b731f4-kube-api-access-bhgsf\") pod \"6665ed83-8dd6-4762-ad16-0b80d2b731f4\" (UID: \"6665ed83-8dd6-4762-ad16-0b80d2b731f4\") "
Nov 03 00:15:22.159291 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:22.159269    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6665ed83-8dd6-4762-ad16-0b80d2b731f4-plugins-config-volume" pod "6665ed83-8dd6-4762-ad16-0b80d2b731f4" (UID: "6665ed83-8dd6-4762-ad16-0b80d2b731f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:22.159455 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:22.159439    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6665ed83-8dd6-4762-ad16-0b80d2b731f4-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:15:22.161044 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:22.161016    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6665ed83-8dd6-4762-ad16-0b80d2b731f4-kube-api-access-bhgsf" pod "6665ed83-8dd6-4762-ad16-0b80d2b731f4" (UID: "6665ed83-8dd6-4762-ad16-0b80d2b731f4"). InnerVolumeSpecName "kube-api-access-bhgsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:22.260990 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:22.260960    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhgsf\" (UniqueName: \"kubernetes.io/projected/6665ed83-8dd6-4762-ad16-0b80d2b731f4-kube-api-access-bhgsf\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:15:22.548297 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:22.548263    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d4c3bf2-7d02-496e-ad2e-665266c9bc1d" path="/var/lib/kubelet/pods/2d4c3bf2-7d02-496e-ad2e-665266c9bc1d/volumes"
Nov 03 00:15:22.631778 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:22.631743    4294 generic.go:356] "Generic (PLEG): container finished" podID="6665ed83-8dd6-4762-ad16-0b80d2b731f4" containerID="f6c249e382e9710d5e33bf2cada1ecf8203db32fedc3253988e2172247575e0c" exitCode=0
Nov 03 00:15:22.631931 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:22.631786    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg"
Nov 03 00:15:22.631931 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:22.631824    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg" event={"ID":"6665ed83-8dd6-4762-ad16-0b80d2b731f4","Type":"ContainerDied","Data":"f6c249e382e9710d5e33bf2cada1ecf8203db32fedc3253988e2172247575e0c"}
Nov 03 00:15:22.631931 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:22.631869    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg" event={"ID":"6665ed83-8dd6-4762-ad16-0b80d2b731f4","Type":"ContainerDied","Data":"9a5e773c6d8bf6b2947cbedf8a476b32971734b385f52ef805ff2ac043d5c1b1"}
Nov 03 00:15:22.631931 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:22.631890    4294 scope.go:122] "RemoveContainer" containerID="f6c249e382e9710d5e33bf2cada1ecf8203db32fedc3253988e2172247575e0c"
Nov 03 00:15:22.640936 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:22.640915    4294 scope.go:122] "RemoveContainer" containerID="f6c249e382e9710d5e33bf2cada1ecf8203db32fedc3253988e2172247575e0c"
Nov 03 00:15:22.641249 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:15:22.641226    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c249e382e9710d5e33bf2cada1ecf8203db32fedc3253988e2172247575e0c\": container with ID starting with f6c249e382e9710d5e33bf2cada1ecf8203db32fedc3253988e2172247575e0c not found: ID does not exist" containerID="f6c249e382e9710d5e33bf2cada1ecf8203db32fedc3253988e2172247575e0c"
Nov 03 00:15:22.641315 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:22.641260    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c249e382e9710d5e33bf2cada1ecf8203db32fedc3253988e2172247575e0c"} err="failed to get container status \"f6c249e382e9710d5e33bf2cada1ecf8203db32fedc3253988e2172247575e0c\": rpc error: code = NotFound desc = could not find container \"f6c249e382e9710d5e33bf2cada1ecf8203db32fedc3253988e2172247575e0c\": container with ID starting with f6c249e382e9710d5e33bf2cada1ecf8203db32fedc3253988e2172247575e0c not found: ID does not exist"
Nov 03 00:15:22.788476 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:22.788445    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg"]
Nov 03 00:15:22.835620 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:22.835547    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c5148c-79b8d87d8c-gg6fg"]
Nov 03 00:15:23.434720 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:23.434690    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:24.547281 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:24.547245    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6665ed83-8dd6-4762-ad16-0b80d2b731f4" path="/var/lib/kubelet/pods/6665ed83-8dd6-4762-ad16-0b80d2b731f4/volumes"
Nov 03 00:15:27.633592 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:27.633558    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:27.806739 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:27.806713    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:27.978729 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:27.978651    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:29.212201 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:29.212171    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:29.653062 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:29.653029    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:30.393598 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:30.393543    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:33.145252 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:33.145217    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:35.992315 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:35.992280    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:36.154440 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:36.154408    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:36.314876 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:36.314788    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:37.808331 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:37.808299    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:40.654533 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:40.654503    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:41.091578 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:41.091548    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:41.752211 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:15:41.752166    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:15:41.983719 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:41.983684    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:42.619803 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:42.619757    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:47.013477 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:47.013424    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:47.176313 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:47.176280    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:47.404681 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:47.404575    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26"]
Nov 03 00:15:47.774750 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:47.774702    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:47.930635 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:47.930604    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:48.612625 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:48.612586    4294 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod150d536c-5dbc-4acf-826c-0f19f3c57320"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod150d536c-5dbc-4acf-826c-0f19f3c57320] : Timed out while waiting for systemd to remove kubepods-burstable-pod150d536c_5dbc_4acf_826c_0f19f3c57320.slice"
Nov 03 00:15:48.612625 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:48.612620    4294 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="150d536c-5dbc-4acf-826c-0f19f3c57320" err="failed to delete cgroup paths for [kubepods burstable pod150d536c-5dbc-4acf-826c-0f19f3c57320] : unable to destroy cgroup paths for cgroup [kubepods burstable pod150d536c-5dbc-4acf-826c-0f19f3c57320] : Timed out while waiting for systemd to remove kubepods-burstable-pod150d536c_5dbc_4acf_826c_0f19f3c57320.slice"
Nov 03 00:15:49.681278 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:49.681242    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:51.943872 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:51.943834    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:52.406574 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:52.405644    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26" podUID="7d025cd4-46da-4b22-af00-991525593a47" containerName="payload-processing" containerID="cri-o://04474419fab214ee46463776e5eb4bed0ed2539cf7c80c2a9fad170af167b580" gracePeriod=25
Nov 03 00:15:53.208425 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:53.208395    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:53.373486 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:53.373449    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26" podUID="7d025cd4-46da-4b22-af00-991525593a47" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:15:53.884372 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:53.884291    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:54.140089 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:54.140014    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:56.564294 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:15:56.564261    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:01.088810 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:01.088773    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:03.204183 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:03.204148    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:03.374279 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:03.374238    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26" podUID="7d025cd4-46da-4b22-af00-991525593a47" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:16:05.535917 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:05.535886    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:10.046626 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:10.046592    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:11.749575 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:16:11.749530    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:16:12.685167 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:12.685139    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26"
Nov 03 00:16:12.742484 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:12.742455    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d025cd4-46da-4b22-af00-991525593a47-kube-api-access-vz5wj\" (UniqueName: \"kubernetes.io/projected/7d025cd4-46da-4b22-af00-991525593a47-kube-api-access-vz5wj\") pod \"7d025cd4-46da-4b22-af00-991525593a47\" (UID: \"7d025cd4-46da-4b22-af00-991525593a47\") "
Nov 03 00:16:12.742663 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:12.742507    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d025cd4-46da-4b22-af00-991525593a47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d025cd4-46da-4b22-af00-991525593a47-plugins-config-volume\") pod \"7d025cd4-46da-4b22-af00-991525593a47\" (UID: \"7d025cd4-46da-4b22-af00-991525593a47\") "
Nov 03 00:16:12.742931 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:12.742907    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d025cd4-46da-4b22-af00-991525593a47-plugins-config-volume" pod "7d025cd4-46da-4b22-af00-991525593a47" (UID: "7d025cd4-46da-4b22-af00-991525593a47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:16:12.744810 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:12.744783    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d025cd4-46da-4b22-af00-991525593a47-kube-api-access-vz5wj" pod "7d025cd4-46da-4b22-af00-991525593a47" (UID: "7d025cd4-46da-4b22-af00-991525593a47"). InnerVolumeSpecName "kube-api-access-vz5wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:16:12.843865 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:12.843836    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz5wj\" (UniqueName: \"kubernetes.io/projected/7d025cd4-46da-4b22-af00-991525593a47-kube-api-access-vz5wj\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:12.843865 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:12.843865    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d025cd4-46da-4b22-af00-991525593a47-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:12.932709 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:12.932625    4294 generic.go:356] "Generic (PLEG): container finished" podID="7d025cd4-46da-4b22-af00-991525593a47" containerID="04474419fab214ee46463776e5eb4bed0ed2539cf7c80c2a9fad170af167b580" exitCode=0
Nov 03 00:16:12.932709 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:12.932673    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26"
Nov 03 00:16:12.932709 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:12.932691    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26" event={"ID":"7d025cd4-46da-4b22-af00-991525593a47","Type":"ContainerDied","Data":"04474419fab214ee46463776e5eb4bed0ed2539cf7c80c2a9fad170af167b580"}
Nov 03 00:16:12.932948 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:12.932733    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26" event={"ID":"7d025cd4-46da-4b22-af00-991525593a47","Type":"ContainerDied","Data":"a89ca4b6ff0a46c70f59de8a108bab35f46716bb9761c43a541e091ffb354f22"}
Nov 03 00:16:12.932948 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:12.932756    4294 scope.go:122] "RemoveContainer" containerID="04474419fab214ee46463776e5eb4bed0ed2539cf7c80c2a9fad170af167b580"
Nov 03 00:16:12.943148 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:12.943125    4294 scope.go:122] "RemoveContainer" containerID="04474419fab214ee46463776e5eb4bed0ed2539cf7c80c2a9fad170af167b580"
Nov 03 00:16:12.943409 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:16:12.943389    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04474419fab214ee46463776e5eb4bed0ed2539cf7c80c2a9fad170af167b580\": container with ID starting with 04474419fab214ee46463776e5eb4bed0ed2539cf7c80c2a9fad170af167b580 not found: ID does not exist" containerID="04474419fab214ee46463776e5eb4bed0ed2539cf7c80c2a9fad170af167b580"
Nov 03 00:16:12.943454 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:12.943419    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04474419fab214ee46463776e5eb4bed0ed2539cf7c80c2a9fad170af167b580"} err="failed to get container status \"04474419fab214ee46463776e5eb4bed0ed2539cf7c80c2a9fad170af167b580\": rpc error: code = NotFound desc = could not find container \"04474419fab214ee46463776e5eb4bed0ed2539cf7c80c2a9fad170af167b580\": container with ID starting with 04474419fab214ee46463776e5eb4bed0ed2539cf7c80c2a9fad170af167b580 not found: ID does not exist"
Nov 03 00:16:13.089599 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:13.089568    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26"]
Nov 03 00:16:13.092326 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:13.092304    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-fe63ac-745df9d78c-mst26"]
Nov 03 00:16:14.119601 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:14.119567    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:14.552216 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:14.552184    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d025cd4-46da-4b22-af00-991525593a47" path="/var/lib/kubelet/pods/7d025cd4-46da-4b22-af00-991525593a47/volumes"
Nov 03 00:16:14.553212 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:14.553186    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:19.143872 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:19.143837    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:22.566108 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:22.566072    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:23.174720 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:23.174683    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:24.124192 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:24.124155    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"]
Nov 03 00:16:24.124647 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:24.124370    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7" podUID="484b42ed-574b-4592-887f-1e85faa347ac" containerName="maas-api" containerID="cri-o://007010af3804f06023786c7b309a80ea1271a0464e8a9f8f37aa3b23bc361658" gracePeriod=30
Nov 03 00:16:24.280626 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:24.280594    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:24.384846 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:24.384779    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"
Nov 03 00:16:24.476709 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:24.476674    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/484b42ed-574b-4592-887f-1e85faa347ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/484b42ed-574b-4592-887f-1e85faa347ac-maas-api-tls\") pod \"484b42ed-574b-4592-887f-1e85faa347ac\" (UID: \"484b42ed-574b-4592-887f-1e85faa347ac\") "
Nov 03 00:16:24.476929 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:24.476726    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/484b42ed-574b-4592-887f-1e85faa347ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/484b42ed-574b-4592-887f-1e85faa347ac-metrics-certs\") pod \"484b42ed-574b-4592-887f-1e85faa347ac\" (UID: \"484b42ed-574b-4592-887f-1e85faa347ac\") "
Nov 03 00:16:24.476929 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:24.476795    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/484b42ed-574b-4592-887f-1e85faa347ac-kube-api-access-j6crb\" (UniqueName: \"kubernetes.io/projected/484b42ed-574b-4592-887f-1e85faa347ac-kube-api-access-j6crb\") pod \"484b42ed-574b-4592-887f-1e85faa347ac\" (UID: \"484b42ed-574b-4592-887f-1e85faa347ac\") "
Nov 03 00:16:24.477183 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:24.476942    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/484b42ed-574b-4592-887f-1e85faa347ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/484b42ed-574b-4592-887f-1e85faa347ac-openshift-service-ca\") pod \"484b42ed-574b-4592-887f-1e85faa347ac\" (UID: \"484b42ed-574b-4592-887f-1e85faa347ac\") "
Nov 03 00:16:24.477308 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:24.477281    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/484b42ed-574b-4592-887f-1e85faa347ac-openshift-service-ca" pod "484b42ed-574b-4592-887f-1e85faa347ac" (UID: "484b42ed-574b-4592-887f-1e85faa347ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:16:24.477422 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:24.477406    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/484b42ed-574b-4592-887f-1e85faa347ac-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:24.479015 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:24.478994    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484b42ed-574b-4592-887f-1e85faa347ac-kube-api-access-j6crb" pod "484b42ed-574b-4592-887f-1e85faa347ac" (UID: "484b42ed-574b-4592-887f-1e85faa347ac"). InnerVolumeSpecName "kube-api-access-j6crb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:16:24.479319 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:24.479301    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/484b42ed-574b-4592-887f-1e85faa347ac-maas-api-tls" pod "484b42ed-574b-4592-887f-1e85faa347ac" (UID: "484b42ed-574b-4592-887f-1e85faa347ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:16:24.479402 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:24.479326    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/484b42ed-574b-4592-887f-1e85faa347ac-metrics-certs" pod "484b42ed-574b-4592-887f-1e85faa347ac" (UID: "484b42ed-574b-4592-887f-1e85faa347ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:16:24.578560 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:24.578520    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/484b42ed-574b-4592-887f-1e85faa347ac-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:24.578560 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:24.578554    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/484b42ed-574b-4592-887f-1e85faa347ac-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:24.578560 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:24.578565    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6crb\" (UniqueName: \"kubernetes.io/projected/484b42ed-574b-4592-887f-1e85faa347ac-kube-api-access-j6crb\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:25.011919 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:25.011883    4294 generic.go:356] "Generic (PLEG): container finished" podID="484b42ed-574b-4592-887f-1e85faa347ac" containerID="007010af3804f06023786c7b309a80ea1271a0464e8a9f8f37aa3b23bc361658" exitCode=0
Nov 03 00:16:25.012153 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:25.011937    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"
Nov 03 00:16:25.012153 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:25.011967    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7" event={"ID":"484b42ed-574b-4592-887f-1e85faa347ac","Type":"ContainerDied","Data":"007010af3804f06023786c7b309a80ea1271a0464e8a9f8f37aa3b23bc361658"}
Nov 03 00:16:25.012153 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:25.012049    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7" event={"ID":"484b42ed-574b-4592-887f-1e85faa347ac","Type":"ContainerDied","Data":"ecd62387e8295bf6ec08ed34bd0c7f8a996033f7bbfff0271bd4b838610e7589"}
Nov 03 00:16:25.012153 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:25.012076    4294 scope.go:122] "RemoveContainer" containerID="007010af3804f06023786c7b309a80ea1271a0464e8a9f8f37aa3b23bc361658"
Nov 03 00:16:25.022218 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:25.022191    4294 scope.go:122] "RemoveContainer" containerID="007010af3804f06023786c7b309a80ea1271a0464e8a9f8f37aa3b23bc361658"
Nov 03 00:16:25.022618 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:16:25.022588    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"007010af3804f06023786c7b309a80ea1271a0464e8a9f8f37aa3b23bc361658\": container with ID starting with 007010af3804f06023786c7b309a80ea1271a0464e8a9f8f37aa3b23bc361658 not found: ID does not exist" containerID="007010af3804f06023786c7b309a80ea1271a0464e8a9f8f37aa3b23bc361658"
Nov 03 00:16:25.022736 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:25.022631    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"007010af3804f06023786c7b309a80ea1271a0464e8a9f8f37aa3b23bc361658"} err="failed to get container status \"007010af3804f06023786c7b309a80ea1271a0464e8a9f8f37aa3b23bc361658\": rpc error: code = NotFound desc = could not find container \"007010af3804f06023786c7b309a80ea1271a0464e8a9f8f37aa3b23bc361658\": container with ID starting with 007010af3804f06023786c7b309a80ea1271a0464e8a9f8f37aa3b23bc361658 not found: ID does not exist"
Nov 03 00:16:25.168966 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:25.168928    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"]
Nov 03 00:16:25.216196 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:25.216164    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-fe63ac-66c48d67dd-xx8r7"]
Nov 03 00:16:25.914516 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:25.914481    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:26.548615 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:26.548578    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="484b42ed-574b-4592-887f-1e85faa347ac" path="/var/lib/kubelet/pods/484b42ed-574b-4592-887f-1e85faa347ac/volumes"
Nov 03 00:16:26.754540 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:26.754501    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:26.932172 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:26.932089    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:28.334568 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:28.334536    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:28.798036 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:28.797975    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:29.506356 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:29.506319    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:30.998152 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:30.998117    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:31.168475 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:31.168441    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:32.447034 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:32.446975    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:32.922536 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:32.922477    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:33.374712 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:33.374671    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:35.205517 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:35.205482    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"]
Nov 03 00:16:35.206018 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:35.205694    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94" podUID="b42173f0-7dde-4729-b3ae-ba90082ca6d8" containerName="maas-api" containerID="cri-o://450153df7da90af38c8bd61e559f86aa544e6a03baffad7fd7173e09c4fe6d23" gracePeriod=30
Nov 03 00:16:35.344933 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:35.344902    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:35.464468 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:35.464444    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"
Nov 03 00:16:35.514267 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:35.514241    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42173f0-7dde-4729-b3ae-ba90082ca6d8-kube-api-access-5jdlc\" (UniqueName: \"kubernetes.io/projected/b42173f0-7dde-4729-b3ae-ba90082ca6d8-kube-api-access-5jdlc\") pod \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\" (UID: \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\") "
Nov 03 00:16:35.514395 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:35.514293    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b42173f0-7dde-4729-b3ae-ba90082ca6d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b42173f0-7dde-4729-b3ae-ba90082ca6d8-maas-api-tls\") pod \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\" (UID: \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\") "
Nov 03 00:16:35.514395 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:35.514365    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b42173f0-7dde-4729-b3ae-ba90082ca6d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42173f0-7dde-4729-b3ae-ba90082ca6d8-openshift-service-ca\") pod \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\" (UID: \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\") "
Nov 03 00:16:35.514525 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:35.514477    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b42173f0-7dde-4729-b3ae-ba90082ca6d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b42173f0-7dde-4729-b3ae-ba90082ca6d8-metrics-certs\") pod \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\" (UID: \"b42173f0-7dde-4729-b3ae-ba90082ca6d8\") "
Nov 03 00:16:35.514787 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:35.514756    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42173f0-7dde-4729-b3ae-ba90082ca6d8-openshift-service-ca" pod "b42173f0-7dde-4729-b3ae-ba90082ca6d8" (UID: "b42173f0-7dde-4729-b3ae-ba90082ca6d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:16:35.515019 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:35.514968    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b42173f0-7dde-4729-b3ae-ba90082ca6d8-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:35.516396 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:35.516371    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42173f0-7dde-4729-b3ae-ba90082ca6d8-kube-api-access-5jdlc" pod "b42173f0-7dde-4729-b3ae-ba90082ca6d8" (UID: "b42173f0-7dde-4729-b3ae-ba90082ca6d8"). InnerVolumeSpecName "kube-api-access-5jdlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:16:35.516638 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:35.516604    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b42173f0-7dde-4729-b3ae-ba90082ca6d8-metrics-certs" pod "b42173f0-7dde-4729-b3ae-ba90082ca6d8" (UID: "b42173f0-7dde-4729-b3ae-ba90082ca6d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:16:35.516724 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:35.516661    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b42173f0-7dde-4729-b3ae-ba90082ca6d8-maas-api-tls" pod "b42173f0-7dde-4729-b3ae-ba90082ca6d8" (UID: "b42173f0-7dde-4729-b3ae-ba90082ca6d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:16:35.616306 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:35.616272    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b42173f0-7dde-4729-b3ae-ba90082ca6d8-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:35.616306 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:35.616304    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jdlc\" (UniqueName: \"kubernetes.io/projected/b42173f0-7dde-4729-b3ae-ba90082ca6d8-kube-api-access-5jdlc\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:35.616520 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:35.616319    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b42173f0-7dde-4729-b3ae-ba90082ca6d8-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:35.756163 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:35.756125    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:35.919485 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:35.919444    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:36.092607 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:36.092517    4294 generic.go:356] "Generic (PLEG): container finished" podID="b42173f0-7dde-4729-b3ae-ba90082ca6d8" containerID="450153df7da90af38c8bd61e559f86aa544e6a03baffad7fd7173e09c4fe6d23" exitCode=0
Nov 03 00:16:36.092607 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:36.092563    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94" event={"ID":"b42173f0-7dde-4729-b3ae-ba90082ca6d8","Type":"ContainerDied","Data":"450153df7da90af38c8bd61e559f86aa544e6a03baffad7fd7173e09c4fe6d23"}
Nov 03 00:16:36.092607 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:36.092569    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"
Nov 03 00:16:36.092890 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:36.092610    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94" event={"ID":"b42173f0-7dde-4729-b3ae-ba90082ca6d8","Type":"ContainerDied","Data":"989279a5162c5ec69a07effe1afae0fb1495d1e34e62f81078ebab53ebcf8485"}
Nov 03 00:16:36.092890 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:36.092634    4294 scope.go:122] "RemoveContainer" containerID="450153df7da90af38c8bd61e559f86aa544e6a03baffad7fd7173e09c4fe6d23"
Nov 03 00:16:36.103091 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:36.103076    4294 scope.go:122] "RemoveContainer" containerID="450153df7da90af38c8bd61e559f86aa544e6a03baffad7fd7173e09c4fe6d23"
Nov 03 00:16:36.103343 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:16:36.103322    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"450153df7da90af38c8bd61e559f86aa544e6a03baffad7fd7173e09c4fe6d23\": container with ID starting with 450153df7da90af38c8bd61e559f86aa544e6a03baffad7fd7173e09c4fe6d23 not found: ID does not exist" containerID="450153df7da90af38c8bd61e559f86aa544e6a03baffad7fd7173e09c4fe6d23"
Nov 03 00:16:36.103394 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:36.103351    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450153df7da90af38c8bd61e559f86aa544e6a03baffad7fd7173e09c4fe6d23"} err="failed to get container status \"450153df7da90af38c8bd61e559f86aa544e6a03baffad7fd7173e09c4fe6d23\": rpc error: code = NotFound desc = could not find container \"450153df7da90af38c8bd61e559f86aa544e6a03baffad7fd7173e09c4fe6d23\": container with ID starting with 450153df7da90af38c8bd61e559f86aa544e6a03baffad7fd7173e09c4fe6d23 not found: ID does not exist"
Nov 03 00:16:36.221894 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:36.221859    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:36.253073 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:36.253044    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"]
Nov 03 00:16:36.255821 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:36.255796    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c25940-f67494564-lqp94"]
Nov 03 00:16:36.549102 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:36.549071    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b42173f0-7dde-4729-b3ae-ba90082ca6d8" path="/var/lib/kubelet/pods/b42173f0-7dde-4729-b3ae-ba90082ca6d8/volumes"
Nov 03 00:16:37.321260 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:37.321217    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:38.314497 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:38.314465    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"]
Nov 03 00:16:38.430852 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:38.430785    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:41.638035 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:41.637995    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:41.754852 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:16:41.754782    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:16:41.806749 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:41.806717    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:42.008737 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:42.008702    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:43.143139 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:43.143106    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:43.608906 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:43.608873    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"]
Nov 03 00:16:43.609124 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:43.609101    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb" podUID="30dbc36b-dc80-47c8-a95a-5ad407f121a1" containerName="istio-proxy" containerID="cri-o://6080a5cf2e47e6e77e9614082203d545a4a9dc1cccc789f7d6b95d985c6ca12c" gracePeriod=30
Nov 03 00:16:43.643125 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:43.643097    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:44.075189 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:44.075130    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:44.620782 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:44.620738    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:46.930647 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:46.930603    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:47.717199 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:47.717165    4294 patch_prober.go:29] interesting pod/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:16:47.717406 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:47.717223    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb" podUID="30dbc36b-dc80-47c8-a95a-5ad407f121a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:16:48.862938 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.862907    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 03 00:16:48.971096 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.971003    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-envoy\") pod \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") "
Nov 03 00:16:48.971096 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.971069    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30dbc36b-dc80-47c8-a95a-5ad407f121a1-kube-api-access-fzg7v\" (UniqueName: \"kubernetes.io/projected/30dbc36b-dc80-47c8-a95a-5ad407f121a1-kube-api-access-fzg7v\") pod \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") "
Nov 03 00:16:48.971367 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.971156    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-podinfo\") pod \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") "
Nov 03 00:16:48.971367 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.971197    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-workload-socket\") pod \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") "
Nov 03 00:16:48.971367 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.971254    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-token\" (UniqueName: \"kubernetes.io/projected/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-token\") pod \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") "
Nov 03 00:16:48.971535 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.971427    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istiod-ca-cert\") pod \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") "
Nov 03 00:16:48.971535 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.971493    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-workload-socket" pod "30dbc36b-dc80-47c8-a95a-5ad407f121a1" (UID: "30dbc36b-dc80-47c8-a95a-5ad407f121a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:16:48.971775 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.971742    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-credential-socket\") pod \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") "
Nov 03 00:16:48.971993 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.971958    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-data\") pod \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") "
Nov 03 00:16:48.972076 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.971813    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istiod-ca-cert" pod "30dbc36b-dc80-47c8-a95a-5ad407f121a1" (UID: "30dbc36b-dc80-47c8-a95a-5ad407f121a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:16:48.972076 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.972032    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-workload-certs\") pod \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\" (UID: \"30dbc36b-dc80-47c8-a95a-5ad407f121a1\") "
Nov 03 00:16:48.972291 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.972012    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-credential-socket" pod "30dbc36b-dc80-47c8-a95a-5ad407f121a1" (UID: "30dbc36b-dc80-47c8-a95a-5ad407f121a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:16:48.972385 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.972365    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-workload-certs" pod "30dbc36b-dc80-47c8-a95a-5ad407f121a1" (UID: "30dbc36b-dc80-47c8-a95a-5ad407f121a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:16:48.972445 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.972416    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-data" pod "30dbc36b-dc80-47c8-a95a-5ad407f121a1" (UID: "30dbc36b-dc80-47c8-a95a-5ad407f121a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:16:48.972564 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.972546    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:48.972631 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.972570    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:48.972631 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.972586    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:48.972631 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.972601    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:48.972631 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.972615    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:48.973458 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.973432    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-token" pod "30dbc36b-dc80-47c8-a95a-5ad407f121a1" (UID: "30dbc36b-dc80-47c8-a95a-5ad407f121a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:16:48.973648 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.973618    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30dbc36b-dc80-47c8-a95a-5ad407f121a1-kube-api-access-fzg7v" pod "30dbc36b-dc80-47c8-a95a-5ad407f121a1" (UID: "30dbc36b-dc80-47c8-a95a-5ad407f121a1"). InnerVolumeSpecName "kube-api-access-fzg7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:16:48.973648 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.973619    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-envoy" pod "30dbc36b-dc80-47c8-a95a-5ad407f121a1" (UID: "30dbc36b-dc80-47c8-a95a-5ad407f121a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:16:48.973886 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.973669    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-podinfo" pod "30dbc36b-dc80-47c8-a95a-5ad407f121a1" (UID: "30dbc36b-dc80-47c8-a95a-5ad407f121a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:16:48.996796 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:48.996765    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"]
Nov 03 00:16:49.017372 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:49.017337    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"]
Nov 03 00:16:49.073918 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:49.073880    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:49.073918 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:49.073911    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzg7v\" (UniqueName: \"kubernetes.io/projected/30dbc36b-dc80-47c8-a95a-5ad407f121a1-kube-api-access-fzg7v\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:49.073918 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:49.073921    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:49.074189 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:49.073930    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30dbc36b-dc80-47c8-a95a-5ad407f121a1-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:49.147677 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:49.147646    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:49.189274 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:49.189241    4294 generic.go:356] "Generic (PLEG): container finished" podID="30dbc36b-dc80-47c8-a95a-5ad407f121a1" containerID="6080a5cf2e47e6e77e9614082203d545a4a9dc1cccc789f7d6b95d985c6ca12c" exitCode=0
Nov 03 00:16:49.189506 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:49.189316    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"
Nov 03 00:16:49.189506 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:49.189325    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb" event={"ID":"30dbc36b-dc80-47c8-a95a-5ad407f121a1","Type":"ContainerDied","Data":"6080a5cf2e47e6e77e9614082203d545a4a9dc1cccc789f7d6b95d985c6ca12c"}
Nov 03 00:16:49.189506 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:49.189381    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb" event={"ID":"30dbc36b-dc80-47c8-a95a-5ad407f121a1","Type":"ContainerDied","Data":"8075b9adedd649e7b9e8ec03a10545929a0cd819958c7f949636a1dae344dbe8"}
Nov 03 00:16:49.189506 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:49.189405    4294 scope.go:122] "RemoveContainer" containerID="6080a5cf2e47e6e77e9614082203d545a4a9dc1cccc789f7d6b95d985c6ca12c"
Nov 03 00:16:49.199788 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:49.199768    4294 scope.go:122] "RemoveContainer" containerID="97abd1623f7b267f2e098831e569610e8eef2a932a8321904ca8eabd028014b0"
Nov 03 00:16:49.208944 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:49.208927    4294 scope.go:122] "RemoveContainer" containerID="6080a5cf2e47e6e77e9614082203d545a4a9dc1cccc789f7d6b95d985c6ca12c"
Nov 03 00:16:49.209257 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:16:49.209237    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6080a5cf2e47e6e77e9614082203d545a4a9dc1cccc789f7d6b95d985c6ca12c\": container with ID starting with 6080a5cf2e47e6e77e9614082203d545a4a9dc1cccc789f7d6b95d985c6ca12c not found: ID does not exist" containerID="6080a5cf2e47e6e77e9614082203d545a4a9dc1cccc789f7d6b95d985c6ca12c"
Nov 03 00:16:49.209330 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:49.209267    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6080a5cf2e47e6e77e9614082203d545a4a9dc1cccc789f7d6b95d985c6ca12c"} err="failed to get container status \"6080a5cf2e47e6e77e9614082203d545a4a9dc1cccc789f7d6b95d985c6ca12c\": rpc error: code = NotFound desc = could not find container \"6080a5cf2e47e6e77e9614082203d545a4a9dc1cccc789f7d6b95d985c6ca12c\": container with ID starting with 6080a5cf2e47e6e77e9614082203d545a4a9dc1cccc789f7d6b95d985c6ca12c not found: ID does not exist"
Nov 03 00:16:49.209330 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:49.209286    4294 scope.go:122] "RemoveContainer" containerID="97abd1623f7b267f2e098831e569610e8eef2a932a8321904ca8eabd028014b0"
Nov 03 00:16:49.209562 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:16:49.209543    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97abd1623f7b267f2e098831e569610e8eef2a932a8321904ca8eabd028014b0\": container with ID starting with 97abd1623f7b267f2e098831e569610e8eef2a932a8321904ca8eabd028014b0 not found: ID does not exist" containerID="97abd1623f7b267f2e098831e569610e8eef2a932a8321904ca8eabd028014b0"
Nov 03 00:16:49.209614 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:49.209567    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97abd1623f7b267f2e098831e569610e8eef2a932a8321904ca8eabd028014b0"} err="failed to get container status \"97abd1623f7b267f2e098831e569610e8eef2a932a8321904ca8eabd028014b0\": rpc error: code = NotFound desc = could not find container \"97abd1623f7b267f2e098831e569610e8eef2a932a8321904ca8eabd028014b0\": container with ID starting with 97abd1623f7b267f2e098831e569610e8eef2a932a8321904ca8eabd028014b0 not found: ID does not exist"
Nov 03 00:16:49.368597 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:49.368560    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"]
Nov 03 00:16:49.371678 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:49.371655    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-c25940-openshift-default-6f69b7b84f-2zsgb"]
Nov 03 00:16:50.105149 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:50.105109    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:50.548639 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:50.548601    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30dbc36b-dc80-47c8-a95a-5ad407f121a1" path="/var/lib/kubelet/pods/30dbc36b-dc80-47c8-a95a-5ad407f121a1/volumes"
Nov 03 00:16:50.591273 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:50.591239    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:51.087568 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:51.087537    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7"]
Nov 03 00:16:51.162124 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:51.162089    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj"]
Nov 03 00:16:51.433495 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:51.433414    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:51.599864 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:51.599832    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:51.765725 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:51.765694    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:53.335058 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:53.335010    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb" podUID="aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" containerName="main" containerID="cri-o://b7a188c5191b2db40a987d9ee8261284ecd1cd8b212f142c29a09c67868cc665" gracePeriod=45
Nov 03 00:16:53.599649 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:53.599587    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 03 00:16:53.728010 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:53.727948    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-tmp-dir\") pod \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") "
Nov 03 00:16:53.728212 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:53.728048    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-home\" (UniqueName: \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-home\") pod \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") "
Nov 03 00:16:53.728212 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:53.728078    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-dshm\") pod \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") "
Nov 03 00:16:53.728212 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:53.728153    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-kube-api-access-mpfj6\" (UniqueName: \"kubernetes.io/projected/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-kube-api-access-mpfj6\") pod \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") "
Nov 03 00:16:53.728212 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:53.728184    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-model-cache\") pod \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") "
Nov 03 00:16:53.728471 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:53.728247    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-tmp-dir" pod "aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" (UID: "aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:16:53.728471 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:53.728286    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-tls-certs\") pod \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\" (UID: \"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a\") "
Nov 03 00:16:53.728471 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:53.728341    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-home" pod "aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" (UID: "aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:16:53.728636 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:53.728546    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-model-cache" pod "aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" (UID: "aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:16:53.728840 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:53.728780    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:53.728840 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:53.728810    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:53.728840 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:53.728824    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:53.730465 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:53.730436    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-tls-certs" pod "aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" (UID: "aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:16:53.730659 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:53.730634    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-dshm" pod "aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" (UID: "aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:16:53.730787 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:53.730769    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-kube-api-access-mpfj6" pod "aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" (UID: "aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a"). InnerVolumeSpecName "kube-api-access-mpfj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:16:53.829921 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:53.829887    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpfj6\" (UniqueName: \"kubernetes.io/projected/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-kube-api-access-mpfj6\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:53.829921 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:53.829915    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:53.829921 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:53.829926    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:16:54.222807 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:54.222769    4294 generic.go:356] "Generic (PLEG): container finished" podID="aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" containerID="b7a188c5191b2db40a987d9ee8261284ecd1cd8b212f142c29a09c67868cc665" exitCode=0
Nov 03 00:16:54.223028 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:54.222832    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"
Nov 03 00:16:54.223028 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:54.222957    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb" event={"ID":"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a","Type":"ContainerDied","Data":"b7a188c5191b2db40a987d9ee8261284ecd1cd8b212f142c29a09c67868cc665"}
Nov 03 00:16:54.223028 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:54.222999    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb" event={"ID":"aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a","Type":"ContainerDied","Data":"a642a18dec55066c521a7274f0125e2533dcdea153678b75568b3df0495571e9"}
Nov 03 00:16:54.223028 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:54.223020    4294 scope.go:122] "RemoveContainer" containerID="b7a188c5191b2db40a987d9ee8261284ecd1cd8b212f142c29a09c67868cc665"
Nov 03 00:16:54.232990 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:54.232960    4294 scope.go:122] "RemoveContainer" containerID="b7a188c5191b2db40a987d9ee8261284ecd1cd8b212f142c29a09c67868cc665"
Nov 03 00:16:54.233290 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:16:54.233267    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7a188c5191b2db40a987d9ee8261284ecd1cd8b212f142c29a09c67868cc665\": container with ID starting with b7a188c5191b2db40a987d9ee8261284ecd1cd8b212f142c29a09c67868cc665 not found: ID does not exist" containerID="b7a188c5191b2db40a987d9ee8261284ecd1cd8b212f142c29a09c67868cc665"
Nov 03 00:16:54.233395 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:54.233297    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7a188c5191b2db40a987d9ee8261284ecd1cd8b212f142c29a09c67868cc665"} err="failed to get container status \"b7a188c5191b2db40a987d9ee8261284ecd1cd8b212f142c29a09c67868cc665\": rpc error: code = NotFound desc = could not find container \"b7a188c5191b2db40a987d9ee8261284ecd1cd8b212f142c29a09c67868cc665\": container with ID starting with b7a188c5191b2db40a987d9ee8261284ecd1cd8b212f142c29a09c67868cc665 not found: ID does not exist"
Nov 03 00:16:54.416823 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:54.416787    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"]
Nov 03 00:16:54.420059 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:54.420037    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb"]
Nov 03 00:16:54.548030 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:54.547971    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" path="/var/lib/kubelet/pods/aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a/volumes"
Nov 03 00:16:54.591813 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:54.591773    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ba1918/premium-w2-ba1918-kserve-85864b4646-nxprb" podUID="aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:16:56.088228 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:56.088177    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7" podUID="959479c1-5aad-4927-9d79-384ef72ebf9e" containerName="payload-processing" containerID="cri-o://9d6bb2e39dd7b760a297d506b011c6395b8fe234a087751b4eb88f7bd21c3b49" gracePeriod=25
Nov 03 00:16:56.103478 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:56.103447    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:56.163455 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:56.163383    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj" podUID="924707d4-bd50-4ce6-9d59-b9abf9d41f64" containerName="payload-pre-processing" containerID="cri-o://12aadbb0dc40ddd4e4ec2d9c212d1d7b8cb08a304c38c7735a4b2ab95ed7e8f4" gracePeriod=25
Nov 03 00:16:56.468835 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:56.468757    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:56.621814 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:56.621781    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:59.976317 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:59.976273    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj" podUID="924707d4-bd50-4ce6-9d59-b9abf9d41f64" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:16:59.976317 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:16:59.976309    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7" podUID="959479c1-5aad-4927-9d79-384ef72ebf9e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:17:01.037279 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:01.037244    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:03.901324 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:03.901289    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:04.017064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.017028    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh" podUID="e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" containerName="main" containerID="cri-o://78183a5d4d1d1598cca5c36c80f77997391b94d96e0cd51818859b4aca2f60fe" gracePeriod=45
Nov 03 00:17:04.037196 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.037165    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf" podUID="2f110c17-1134-4f5c-b9a6-51abb2ac481f" containerName="main" containerID="cri-o://e612f806cc9137142b79e598c2ad6c2e82273cae6b39de891d25a1047e05c7c4" gracePeriod=45
Nov 03 00:17:04.296271 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.296244    4294 generic.go:356] "Generic (PLEG): container finished" podID="2f110c17-1134-4f5c-b9a6-51abb2ac481f" containerID="e612f806cc9137142b79e598c2ad6c2e82273cae6b39de891d25a1047e05c7c4" exitCode=0
Nov 03 00:17:04.296413 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.296342    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf" event={"ID":"2f110c17-1134-4f5c-b9a6-51abb2ac481f","Type":"ContainerDied","Data":"e612f806cc9137142b79e598c2ad6c2e82273cae6b39de891d25a1047e05c7c4"}
Nov 03 00:17:04.297876 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.297855    4294 generic.go:356] "Generic (PLEG): container finished" podID="e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" containerID="78183a5d4d1d1598cca5c36c80f77997391b94d96e0cd51818859b4aca2f60fe" exitCode=0
Nov 03 00:17:04.298031 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.297898    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh" event={"ID":"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6","Type":"ContainerDied","Data":"78183a5d4d1d1598cca5c36c80f77997391b94d96e0cd51818859b4aca2f60fe"}
Nov 03 00:17:04.328374 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.328345    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 03 00:17:04.330201 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.330185    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 03 00:17:04.351266 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.351230    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-home\" (UniqueName: \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-home\") pod \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") "
Nov 03 00:17:04.351428 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.351327    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-home\" (UniqueName: \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-home\") pod \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") "
Nov 03 00:17:04.351428 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.351369    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-tls-certs\") pod \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") "
Nov 03 00:17:04.351581 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.351446    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-tmp-dir\") pod \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") "
Nov 03 00:17:04.351581 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.351476    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-model-cache\") pod \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") "
Nov 03 00:17:04.351581 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.351514    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-home" pod "e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" (UID: "e530f6fc-adcd-402a-b7e8-64d7ee34a7a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:17:04.351581 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.351563    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f110c17-1134-4f5c-b9a6-51abb2ac481f-kube-api-access-m6kq6\" (UniqueName: \"kubernetes.io/projected/2f110c17-1134-4f5c-b9a6-51abb2ac481f-kube-api-access-m6kq6\") pod \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") "
Nov 03 00:17:04.351815 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.351604    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-kube-api-access-w7wbj\" (UniqueName: \"kubernetes.io/projected/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-kube-api-access-w7wbj\") pod \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") "
Nov 03 00:17:04.351815 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.351667    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f110c17-1134-4f5c-b9a6-51abb2ac481f-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f110c17-1134-4f5c-b9a6-51abb2ac481f-tls-certs\") pod \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") "
Nov 03 00:17:04.351815 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.351641    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-home" pod "2f110c17-1134-4f5c-b9a6-51abb2ac481f" (UID: "2f110c17-1134-4f5c-b9a6-51abb2ac481f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:17:04.351815 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.351702    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-dshm\") pod \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") "
Nov 03 00:17:04.351815 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.351778    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-tmp-dir\") pod \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") "
Nov 03 00:17:04.351815 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.351722    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-tmp-dir" pod "e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" (UID: "e530f6fc-adcd-402a-b7e8-64d7ee34a7a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:17:04.352199 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.351831    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-model-cache\") pod \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\" (UID: \"2f110c17-1134-4f5c-b9a6-51abb2ac481f\") "
Nov 03 00:17:04.352199 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.351859    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-dshm\") pod \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\" (UID: \"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6\") "
Nov 03 00:17:04.352199 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.351939    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-model-cache" pod "e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" (UID: "e530f6fc-adcd-402a-b7e8-64d7ee34a7a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:17:04.352199 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.352171    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-tmp-dir" pod "2f110c17-1134-4f5c-b9a6-51abb2ac481f" (UID: "2f110c17-1134-4f5c-b9a6-51abb2ac481f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:17:04.352446 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.352366    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-model-cache" pod "2f110c17-1134-4f5c-b9a6-51abb2ac481f" (UID: "2f110c17-1134-4f5c-b9a6-51abb2ac481f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:17:04.352520 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.352440    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:04.352520 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.352461    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:04.352520 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.352476    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:04.352520 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.352492    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:04.352520 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.352505    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:04.354495 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.354465    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-dshm" pod "e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" (UID: "e530f6fc-adcd-402a-b7e8-64d7ee34a7a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:17:04.354615 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.354581    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-kube-api-access-w7wbj" pod "e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" (UID: "e530f6fc-adcd-402a-b7e8-64d7ee34a7a6"). InnerVolumeSpecName "kube-api-access-w7wbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:17:04.354615 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.354604    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-tls-certs" pod "e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" (UID: "e530f6fc-adcd-402a-b7e8-64d7ee34a7a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:17:04.354734 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.354717    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f110c17-1134-4f5c-b9a6-51abb2ac481f-tls-certs" pod "2f110c17-1134-4f5c-b9a6-51abb2ac481f" (UID: "2f110c17-1134-4f5c-b9a6-51abb2ac481f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:17:04.354819 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.354797    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f110c17-1134-4f5c-b9a6-51abb2ac481f-kube-api-access-m6kq6" pod "2f110c17-1134-4f5c-b9a6-51abb2ac481f" (UID: "2f110c17-1134-4f5c-b9a6-51abb2ac481f"). InnerVolumeSpecName "kube-api-access-m6kq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:17:04.354884 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.354840    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-dshm" pod "2f110c17-1134-4f5c-b9a6-51abb2ac481f" (UID: "2f110c17-1134-4f5c-b9a6-51abb2ac481f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:17:04.398804 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.398771    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"]
Nov 03 00:17:04.453615 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.453536    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f110c17-1134-4f5c-b9a6-51abb2ac481f-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:04.453615 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.453571    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:04.453615 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.453586    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f110c17-1134-4f5c-b9a6-51abb2ac481f-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:04.453615 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.453599    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:04.453615 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.453611    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:04.453959 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.453624    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6kq6\" (UniqueName: \"kubernetes.io/projected/2f110c17-1134-4f5c-b9a6-51abb2ac481f-kube-api-access-m6kq6\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:04.453959 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.453639    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7wbj\" (UniqueName: \"kubernetes.io/projected/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6-kube-api-access-w7wbj\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:04.462310 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.462276    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:04.889561 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:04.889518    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:05.304064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:05.304032    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"
Nov 03 00:17:05.304064 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:05.304049    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf" event={"ID":"2f110c17-1134-4f5c-b9a6-51abb2ac481f","Type":"ContainerDied","Data":"664f183492a9e7859f7ca1873a2117c0668959b99eec0d4b6686ba4d61c5bb1a"}
Nov 03 00:17:05.304564 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:05.304099    4294 scope.go:122] "RemoveContainer" containerID="e612f806cc9137142b79e598c2ad6c2e82273cae6b39de891d25a1047e05c7c4"
Nov 03 00:17:05.305535 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:05.305517    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"
Nov 03 00:17:05.305646 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:05.305562    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh" event={"ID":"e530f6fc-adcd-402a-b7e8-64d7ee34a7a6","Type":"ContainerDied","Data":"da59dfdc50565b54aec988cbd46cc38a5d3fb2351d6825a100d97e8a5523b491"}
Nov 03 00:17:05.313391 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:05.313374    4294 scope.go:122] "RemoveContainer" containerID="78183a5d4d1d1598cca5c36c80f77997391b94d96e0cd51818859b4aca2f60fe"
Nov 03 00:17:05.461760 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:05.461706    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"]
Nov 03 00:17:05.465481 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:05.465442    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/distinct-w1-3d8404-kserve-549fc4f59f-wfmxh"]
Nov 03 00:17:05.613158 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:05.613068    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"]
Nov 03 00:17:05.616476 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:05.616450    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-3d8404/simulator-w1-3d8404-kserve-6d6477c969-hz6jf"]
Nov 03 00:17:05.780569 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:05.780540    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:06.529519 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:06.529483    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh"]
Nov 03 00:17:06.557605 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:06.557563    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f110c17-1134-4f5c-b9a6-51abb2ac481f" path="/var/lib/kubelet/pods/2f110c17-1134-4f5c-b9a6-51abb2ac481f/volumes"
Nov 03 00:17:06.558188 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:06.558164    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" path="/var/lib/kubelet/pods/e530f6fc-adcd-402a-b7e8-64d7ee34a7a6/volumes"
Nov 03 00:17:06.632661 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:06.632630    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f"]
Nov 03 00:17:06.900305 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:06.900215    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:07.052372 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:07.052342    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:09.590773 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.590725    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:09.866443 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.866370    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7427-e089fb"]
Nov 03 00:17:09.866877 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.866865    4294 cpu_manager.go:414] "containerMap: removing container" podUID="484b42ed-574b-4592-887f-1e85faa347ac" containerName="maas-api"
Nov 03 00:17:09.866922 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.866880    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="484b42ed-574b-4592-887f-1e85faa347ac" containerName="maas-api"
Nov 03 00:17:09.866922 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.866893    4294 cpu_manager.go:414] "containerMap: removing container" podUID="30dbc36b-dc80-47c8-a95a-5ad407f121a1" containerName="istio-proxy"
Nov 03 00:17:09.866922 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.866899    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30dbc36b-dc80-47c8-a95a-5ad407f121a1" containerName="istio-proxy"
Nov 03 00:17:09.866922 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.866907    4294 cpu_manager.go:414] "containerMap: removing container" podUID="2f110c17-1134-4f5c-b9a6-51abb2ac481f" containerName="main"
Nov 03 00:17:09.866922 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.866912    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f110c17-1134-4f5c-b9a6-51abb2ac481f" containerName="main"
Nov 03 00:17:09.866922 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.866920    4294 cpu_manager.go:414] "containerMap: removing container" podUID="2d4c3bf2-7d02-496e-ad2e-665266c9bc1d" containerName="cleanup"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.866924    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d4c3bf2-7d02-496e-ad2e-665266c9bc1d" containerName="cleanup"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.866935    4294 cpu_manager.go:414] "containerMap: removing container" podUID="2d4c3bf2-7d02-496e-ad2e-665266c9bc1d" containerName="cleanup"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.866940    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d4c3bf2-7d02-496e-ad2e-665266c9bc1d" containerName="cleanup"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.866949    4294 cpu_manager.go:414] "containerMap: removing container" podUID="7d025cd4-46da-4b22-af00-991525593a47" containerName="payload-processing"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.866954    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d025cd4-46da-4b22-af00-991525593a47" containerName="payload-processing"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.866962    4294 cpu_manager.go:414] "containerMap: removing container" podUID="4df8f166-1d50-4ff1-9965-43d332f18d63" containerName="cleanup"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.866968    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df8f166-1d50-4ff1-9965-43d332f18d63" containerName="cleanup"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.866974    4294 cpu_manager.go:414] "containerMap: removing container" podUID="4df8f166-1d50-4ff1-9965-43d332f18d63" containerName="cleanup"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.866990    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df8f166-1d50-4ff1-9965-43d332f18d63" containerName="cleanup"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.866999    4294 cpu_manager.go:414] "containerMap: removing container" podUID="150d536c-5dbc-4acf-826c-0f19f3c57320" containerName="cleanup"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867003    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150d536c-5dbc-4acf-826c-0f19f3c57320" containerName="cleanup"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867009    4294 cpu_manager.go:414] "containerMap: removing container" podUID="aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" containerName="main"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867015    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" containerName="main"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867021    4294 cpu_manager.go:414] "containerMap: removing container" podUID="6665ed83-8dd6-4762-ad16-0b80d2b731f4" containerName="payload-pre-processing"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867026    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6665ed83-8dd6-4762-ad16-0b80d2b731f4" containerName="payload-pre-processing"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867035    4294 cpu_manager.go:414] "containerMap: removing container" podUID="2d4c3bf2-7d02-496e-ad2e-665266c9bc1d" containerName="cleanup"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867040    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d4c3bf2-7d02-496e-ad2e-665266c9bc1d" containerName="cleanup"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867046    4294 cpu_manager.go:414] "containerMap: removing container" podUID="30dbc36b-dc80-47c8-a95a-5ad407f121a1" containerName="istio-proxy"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867050    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30dbc36b-dc80-47c8-a95a-5ad407f121a1" containerName="istio-proxy"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867059    4294 cpu_manager.go:414] "containerMap: removing container" podUID="4d799ed5-6504-4440-aa0f-84ba25128c25" containerName="maas-api"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867064    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d799ed5-6504-4440-aa0f-84ba25128c25" containerName="maas-api"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867080    4294 cpu_manager.go:414] "containerMap: removing container" podUID="e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" containerName="main"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867085    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" containerName="main"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867093    4294 cpu_manager.go:414] "containerMap: removing container" podUID="150d536c-5dbc-4acf-826c-0f19f3c57320" containerName="cleanup"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867097    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150d536c-5dbc-4acf-826c-0f19f3c57320" containerName="cleanup"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867103    4294 cpu_manager.go:414] "containerMap: removing container" podUID="b42173f0-7dde-4729-b3ae-ba90082ca6d8" containerName="maas-api"
Nov 03 00:17:09.867138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867107    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b42173f0-7dde-4729-b3ae-ba90082ca6d8" containerName="maas-api"
Nov 03 00:17:09.868023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867181    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df8f166-1d50-4ff1-9965-43d332f18d63" containerName="cleanup"
Nov 03 00:17:09.868023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867189    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e530f6fc-adcd-402a-b7e8-64d7ee34a7a6" containerName="main"
Nov 03 00:17:09.868023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867198    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="150d536c-5dbc-4acf-826c-0f19f3c57320" containerName="cleanup"
Nov 03 00:17:09.868023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867203    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="b42173f0-7dde-4729-b3ae-ba90082ca6d8" containerName="maas-api"
Nov 03 00:17:09.868023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867209    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="484b42ed-574b-4592-887f-1e85faa347ac" containerName="maas-api"
Nov 03 00:17:09.868023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867215    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="30dbc36b-dc80-47c8-a95a-5ad407f121a1" containerName="istio-proxy"
Nov 03 00:17:09.868023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867220    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f110c17-1134-4f5c-b9a6-51abb2ac481f" containerName="main"
Nov 03 00:17:09.868023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867225    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d4c3bf2-7d02-496e-ad2e-665266c9bc1d" containerName="cleanup"
Nov 03 00:17:09.868023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867231    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d4c3bf2-7d02-496e-ad2e-665266c9bc1d" containerName="cleanup"
Nov 03 00:17:09.868023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867237    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d025cd4-46da-4b22-af00-991525593a47" containerName="payload-processing"
Nov 03 00:17:09.868023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867243    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df8f166-1d50-4ff1-9965-43d332f18d63" containerName="cleanup"
Nov 03 00:17:09.868023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867249    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df8f166-1d50-4ff1-9965-43d332f18d63" containerName="cleanup"
Nov 03 00:17:09.868023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867255    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="150d536c-5dbc-4acf-826c-0f19f3c57320" containerName="cleanup"
Nov 03 00:17:09.868023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867260    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4f12b9-2c3e-46a1-9e99-d0dce3ceee8a" containerName="main"
Nov 03 00:17:09.868023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867265    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6665ed83-8dd6-4762-ad16-0b80d2b731f4" containerName="payload-pre-processing"
Nov 03 00:17:09.868023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.867272    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d799ed5-6504-4440-aa0f-84ba25128c25" containerName="maas-api"
Nov 03 00:17:09.871759 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.871734    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7427-e089fb"
Nov 03 00:17:09.912040 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.912003    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clgn4\" (UniqueName: \"kubernetes.io/projected/3da850c4-ee0b-4a88-a421-7f8800ba06e2-kube-api-access-clgn4\") pod \"test-curl-7427-e089fb\" (UID: \"3da850c4-ee0b-4a88-a421-7f8800ba06e2\") " pod="opendatahub/test-curl-7427-e089fb"
Nov 03 00:17:09.919181 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.919154    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7427-e089fb"]
Nov 03 00:17:09.963815 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.963783    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wbwlb\""
Nov 03 00:17:09.977135 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.977092    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj" podUID="924707d4-bd50-4ce6-9d59-b9abf9d41f64" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:17:09.977497 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:09.977464    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7" podUID="959479c1-5aad-4927-9d79-384ef72ebf9e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:17:10.012872 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:10.012833    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clgn4\" (UniqueName: \"kubernetes.io/projected/3da850c4-ee0b-4a88-a421-7f8800ba06e2-kube-api-access-clgn4\") pod \"test-curl-7427-e089fb\" (UID: \"3da850c4-ee0b-4a88-a421-7f8800ba06e2\") " pod="opendatahub/test-curl-7427-e089fb"
Nov 03 00:17:10.066855 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:10.066817    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clgn4\" (UniqueName: \"kubernetes.io/projected/3da850c4-ee0b-4a88-a421-7f8800ba06e2-kube-api-access-clgn4\") pod \"test-curl-7427-e089fb\" (UID: \"3da850c4-ee0b-4a88-a421-7f8800ba06e2\") " pod="opendatahub/test-curl-7427-e089fb"
Nov 03 00:17:10.183818 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:10.183721    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7427-e089fb"
Nov 03 00:17:10.550058 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:17:10.550028    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3da850c4_ee0b_4a88_a421_7f8800ba06e2.slice/crio-5a052ef872df4637d29416cd7a7c8370b5a1c913a128a268fb20a8e6c7bea7eb WatchSource:0}: Error finding container 5a052ef872df4637d29416cd7a7c8370b5a1c913a128a268fb20a8e6c7bea7eb: Status 404 returned error can't find the container with id 5a052ef872df4637d29416cd7a7c8370b5a1c913a128a268fb20a8e6c7bea7eb
Nov 03 00:17:10.550058 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:10.550053    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7427-e089fb"]
Nov 03 00:17:10.557797 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:10.557751    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:11.034651 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:11.034614    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq"]
Nov 03 00:17:11.135101 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:11.135067    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76"]
Nov 03 00:17:11.349161 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:11.349071    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7427-e089fb" event={"ID":"3da850c4-ee0b-4a88-a421-7f8800ba06e2","Type":"ContainerStarted","Data":"33236b2692cbbd0df4bc2876b575e804e7bdd59d8830db51a2d474e7c9ea7d03"}
Nov 03 00:17:11.349161 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:11.349122    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7427-e089fb" event={"ID":"3da850c4-ee0b-4a88-a421-7f8800ba06e2","Type":"ContainerStarted","Data":"5a052ef872df4637d29416cd7a7c8370b5a1c913a128a268fb20a8e6c7bea7eb"}
Nov 03 00:17:11.412529 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:11.412495    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:11.530285 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:11.530245    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh" podUID="78e187a5-e905-410b-a2c5-ddced7693786" containerName="payload-processing" containerID="cri-o://de513cd3a4405ddf97a023f3ed2226e1536e7cef74f34cb24b5527d5b6ec96c6" gracePeriod=25
Nov 03 00:17:11.582911 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:11.582875    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:11.632997 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:11.632891    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f" podUID="b716a68d-1d7e-48d6-98a5-6fdc5472cac9" containerName="payload-pre-processing" containerID="cri-o://c422267b001cf69d1eae9ddb63f2283ee6bbd619e80870d35512f03dc0418783" gracePeriod=25
Nov 03 00:17:11.747482 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:11.747448    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:11.750503 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:17:11.750478    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:17:12.934115 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:12.934063    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7427-e089fb" podStartSLOduration=3.9340481499999997 podStartE2EDuration="3.93404815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:17:11.45714319 +0000 UTC m=+5071.025497989" watchObservedRunningTime="2026-11-03 00:17:12.93404815 +0000 UTC m=+5072.502402936"
Nov 03 00:17:12.936816 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:12.936789    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7427-e089fb"]
Nov 03 00:17:12.939577 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:12.939553    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7427-e089fb"]
Nov 03 00:17:13.097135 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:13.097102    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7427-a3ac02"]
Nov 03 00:17:13.097678 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:13.097663    4294 cpu_manager.go:414] "containerMap: removing container" podUID="3da850c4-ee0b-4a88-a421-7f8800ba06e2" containerName="test-curl-7427-e089fb"
Nov 03 00:17:13.097746 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:13.097679    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da850c4-ee0b-4a88-a421-7f8800ba06e2" containerName="test-curl-7427-e089fb"
Nov 03 00:17:13.097746 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:13.097719    4294 cpu_manager.go:414] "containerMap: removing container" podUID="4df8f166-1d50-4ff1-9965-43d332f18d63" containerName="cleanup"
Nov 03 00:17:13.097746 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:13.097725    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df8f166-1d50-4ff1-9965-43d332f18d63" containerName="cleanup"
Nov 03 00:17:13.097746 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:13.097735    4294 cpu_manager.go:414] "containerMap: removing container" podUID="150d536c-5dbc-4acf-826c-0f19f3c57320" containerName="cleanup"
Nov 03 00:17:13.097746 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:13.097740    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150d536c-5dbc-4acf-826c-0f19f3c57320" containerName="cleanup"
Nov 03 00:17:13.097959 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:13.097834    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da850c4-ee0b-4a88-a421-7f8800ba06e2" containerName="test-curl-7427-e089fb"
Nov 03 00:17:13.097959 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:13.097843    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="150d536c-5dbc-4acf-826c-0f19f3c57320" containerName="cleanup"
Nov 03 00:17:13.097959 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:13.097852    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="30dbc36b-dc80-47c8-a95a-5ad407f121a1" containerName="istio-proxy"
Nov 03 00:17:13.097959 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:13.097861    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d4c3bf2-7d02-496e-ad2e-665266c9bc1d" containerName="cleanup"
Nov 03 00:17:13.101230 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:13.101213    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7427-a3ac02"
Nov 03 00:17:13.148949 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:13.148440    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n4gg\" (UniqueName: \"kubernetes.io/projected/b86a51a3-351e-46c3-ad35-68eb39e6ee8d-kube-api-access-7n4gg\") pod \"test-curl-7427-a3ac02\" (UID: \"b86a51a3-351e-46c3-ad35-68eb39e6ee8d\") " pod="opendatahub/test-curl-7427-a3ac02"
Nov 03 00:17:13.151148 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:13.151124    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7427-a3ac02"]
Nov 03 00:17:13.249472 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:13.249439    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n4gg\" (UniqueName: \"kubernetes.io/projected/b86a51a3-351e-46c3-ad35-68eb39e6ee8d-kube-api-access-7n4gg\") pod \"test-curl-7427-a3ac02\" (UID: \"b86a51a3-351e-46c3-ad35-68eb39e6ee8d\") " pod="opendatahub/test-curl-7427-a3ac02"
Nov 03 00:17:13.303582 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:13.303548    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n4gg\" (UniqueName: \"kubernetes.io/projected/b86a51a3-351e-46c3-ad35-68eb39e6ee8d-kube-api-access-7n4gg\") pod \"test-curl-7427-a3ac02\" (UID: \"b86a51a3-351e-46c3-ad35-68eb39e6ee8d\") " pod="opendatahub/test-curl-7427-a3ac02"
Nov 03 00:17:13.360254 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:13.360220    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7427-e089fb" podUID="3da850c4-ee0b-4a88-a421-7f8800ba06e2" containerName="test-curl-7427-e089fb" containerID="cri-o://33236b2692cbbd0df4bc2876b575e804e7bdd59d8830db51a2d474e7c9ea7d03" gracePeriod=2
Nov 03 00:17:13.408973 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:17:13.408934    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7427-e089fb\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="3da850c4-ee0b-4a88-a421-7f8800ba06e2" pod="opendatahub/test-curl-7427-e089fb"
Nov 03 00:17:13.413325 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:13.413306    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7427-a3ac02"
Nov 03 00:17:13.775867 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:13.775838    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7427-a3ac02"]
Nov 03 00:17:13.776874 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:17:13.776847    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb86a51a3_351e_46c3_ad35_68eb39e6ee8d.slice/crio-46297ac7eaa122e34b59e74aef5dc3c5133478391c7f6a701ca6c4d97907daaa WatchSource:0}: Error finding container 46297ac7eaa122e34b59e74aef5dc3c5133478391c7f6a701ca6c4d97907daaa: Status 404 returned error can't find the container with id 46297ac7eaa122e34b59e74aef5dc3c5133478391c7f6a701ca6c4d97907daaa
Nov 03 00:17:14.003999 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:14.003954    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f" podUID="b716a68d-1d7e-48d6-98a5-6fdc5472cac9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:17:14.367142 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:14.367104    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7427-a3ac02" event={"ID":"b86a51a3-351e-46c3-ad35-68eb39e6ee8d","Type":"ContainerStarted","Data":"b31d05b7637f259809114f1bb6cd3971e61b84b08c3150ce11b73e871bd1100f"}
Nov 03 00:17:14.367316 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:14.367153    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7427-a3ac02" event={"ID":"b86a51a3-351e-46c3-ad35-68eb39e6ee8d","Type":"ContainerStarted","Data":"46297ac7eaa122e34b59e74aef5dc3c5133478391c7f6a701ca6c4d97907daaa"}
Nov 03 00:17:14.415761 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:17:14.415725    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7427-e089fb\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="3da850c4-ee0b-4a88-a421-7f8800ba06e2" pod="opendatahub/test-curl-7427-e089fb"
Nov 03 00:17:14.569909 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:14.569858    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7427-a3ac02" podStartSLOduration=1.569843454 podStartE2EDuration="1.569843454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:17:14.52418539 +0000 UTC m=+5074.092540178" watchObservedRunningTime="2026-11-03 00:17:14.569843454 +0000 UTC m=+5074.138198240"
Nov 03 00:17:14.678860 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:14.678774    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:14.850361 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:14.850328    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:15.009126 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:15.009090    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh" podUID="78e187a5-e905-410b-a2c5-ddced7693786" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:17:15.014096 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:15.014066    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:15.583934 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:15.583910    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7427-e089fb"
Nov 03 00:17:15.677503 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:15.677422    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da850c4-ee0b-4a88-a421-7f8800ba06e2-kube-api-access-clgn4\" (UniqueName: \"kubernetes.io/projected/3da850c4-ee0b-4a88-a421-7f8800ba06e2-kube-api-access-clgn4\") pod \"3da850c4-ee0b-4a88-a421-7f8800ba06e2\" (UID: \"3da850c4-ee0b-4a88-a421-7f8800ba06e2\") "
Nov 03 00:17:15.679788 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:15.679759    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da850c4-ee0b-4a88-a421-7f8800ba06e2-kube-api-access-clgn4" pod "3da850c4-ee0b-4a88-a421-7f8800ba06e2" (UID: "3da850c4-ee0b-4a88-a421-7f8800ba06e2"). InnerVolumeSpecName "kube-api-access-clgn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:17:15.778800 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:15.778764    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clgn4\" (UniqueName: \"kubernetes.io/projected/3da850c4-ee0b-4a88-a421-7f8800ba06e2-kube-api-access-clgn4\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:15.983107 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:15.983026    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7427-a3ac02"]
Nov 03 00:17:15.985581 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:15.985551    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7427-a3ac02"]
Nov 03 00:17:16.035306 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.035265    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq" podUID="36cb1d31-8467-441f-90bf-07ba20b0f3c6" containerName="payload-processing" containerID="cri-o://a2f19b1757ca0654957594e93f52980ad5d3a0302b8034a247d47c984c66ac93" gracePeriod=25
Nov 03 00:17:16.136065 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.136003    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76" podUID="95ad5c99-119f-46ca-8664-8f2878dbca94" containerName="payload-pre-processing" containerID="cri-o://f18166f5e45be864f02b4bf751c71292a3084367dcb9f04b632d389284f77dea" gracePeriod=25
Nov 03 00:17:16.270319 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.270292    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:16.382412 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.382376    4294 generic.go:356] "Generic (PLEG): container finished" podID="3da850c4-ee0b-4a88-a421-7f8800ba06e2" containerID="33236b2692cbbd0df4bc2876b575e804e7bdd59d8830db51a2d474e7c9ea7d03" exitCode=137
Nov 03 00:17:16.382578 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.382431    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7427-e089fb"
Nov 03 00:17:16.382578 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.382475    4294 scope.go:122] "RemoveContainer" containerID="33236b2692cbbd0df4bc2876b575e804e7bdd59d8830db51a2d474e7c9ea7d03"
Nov 03 00:17:16.393010 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.392792    4294 generic.go:356] "Generic (PLEG): container finished" podID="924707d4-bd50-4ce6-9d59-b9abf9d41f64" containerID="12aadbb0dc40ddd4e4ec2d9c212d1d7b8cb08a304c38c7735a4b2ab95ed7e8f4" exitCode=0
Nov 03 00:17:16.393010 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.392913    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj" event={"ID":"924707d4-bd50-4ce6-9d59-b9abf9d41f64","Type":"ContainerDied","Data":"12aadbb0dc40ddd4e4ec2d9c212d1d7b8cb08a304c38c7735a4b2ab95ed7e8f4"}
Nov 03 00:17:16.396539 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.396510    4294 generic.go:356] "Generic (PLEG): container finished" podID="959479c1-5aad-4927-9d79-384ef72ebf9e" containerID="9d6bb2e39dd7b760a297d506b011c6395b8fe234a087751b4eb88f7bd21c3b49" exitCode=0
Nov 03 00:17:16.396793 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.396769    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7427-a3ac02" podUID="b86a51a3-351e-46c3-ad35-68eb39e6ee8d" containerName="test-curl-7427-a3ac02" containerID="cri-o://b31d05b7637f259809114f1bb6cd3971e61b84b08c3150ce11b73e871bd1100f" gracePeriod=2
Nov 03 00:17:16.397133 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.397111    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7" event={"ID":"959479c1-5aad-4927-9d79-384ef72ebf9e","Type":"ContainerDied","Data":"9d6bb2e39dd7b760a297d506b011c6395b8fe234a087751b4eb88f7bd21c3b49"}
Nov 03 00:17:16.406272 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.406111    4294 scope.go:122] "RemoveContainer" containerID="33236b2692cbbd0df4bc2876b575e804e7bdd59d8830db51a2d474e7c9ea7d03"
Nov 03 00:17:16.406606 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:17:16.406547    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33236b2692cbbd0df4bc2876b575e804e7bdd59d8830db51a2d474e7c9ea7d03\": container with ID starting with 33236b2692cbbd0df4bc2876b575e804e7bdd59d8830db51a2d474e7c9ea7d03 not found: ID does not exist" containerID="33236b2692cbbd0df4bc2876b575e804e7bdd59d8830db51a2d474e7c9ea7d03"
Nov 03 00:17:16.406606 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.406588    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33236b2692cbbd0df4bc2876b575e804e7bdd59d8830db51a2d474e7c9ea7d03"} err="failed to get container status \"33236b2692cbbd0df4bc2876b575e804e7bdd59d8830db51a2d474e7c9ea7d03\": rpc error: code = NotFound desc = could not find container \"33236b2692cbbd0df4bc2876b575e804e7bdd59d8830db51a2d474e7c9ea7d03\": container with ID starting with 33236b2692cbbd0df4bc2876b575e804e7bdd59d8830db51a2d474e7c9ea7d03 not found: ID does not exist"
Nov 03 00:17:16.422168 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.422146    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:16.426861 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.426843    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7"
Nov 03 00:17:16.441846 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:17:16.441814    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7427-e089fb\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="3da850c4-ee0b-4a88-a421-7f8800ba06e2" pod="opendatahub/test-curl-7427-e089fb"
Nov 03 00:17:16.460949 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.460933    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj"
Nov 03 00:17:16.491400 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.491374    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/959479c1-5aad-4927-9d79-384ef72ebf9e-kube-api-access-x4drb\" (UniqueName: \"kubernetes.io/projected/959479c1-5aad-4927-9d79-384ef72ebf9e-kube-api-access-x4drb\") pod \"959479c1-5aad-4927-9d79-384ef72ebf9e\" (UID: \"959479c1-5aad-4927-9d79-384ef72ebf9e\") "
Nov 03 00:17:16.491515 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.491419    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/959479c1-5aad-4927-9d79-384ef72ebf9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959479c1-5aad-4927-9d79-384ef72ebf9e-plugins-config-volume\") pod \"959479c1-5aad-4927-9d79-384ef72ebf9e\" (UID: \"959479c1-5aad-4927-9d79-384ef72ebf9e\") "
Nov 03 00:17:16.491807 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.491786    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/959479c1-5aad-4927-9d79-384ef72ebf9e-plugins-config-volume" pod "959479c1-5aad-4927-9d79-384ef72ebf9e" (UID: "959479c1-5aad-4927-9d79-384ef72ebf9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:17:16.493466 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.493448    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959479c1-5aad-4927-9d79-384ef72ebf9e-kube-api-access-x4drb" pod "959479c1-5aad-4927-9d79-384ef72ebf9e" (UID: "959479c1-5aad-4927-9d79-384ef72ebf9e"). InnerVolumeSpecName "kube-api-access-x4drb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:17:16.548256 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.548182    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3da850c4-ee0b-4a88-a421-7f8800ba06e2" path="/var/lib/kubelet/pods/3da850c4-ee0b-4a88-a421-7f8800ba06e2/volumes"
Nov 03 00:17:16.592506 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.592475    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924707d4-bd50-4ce6-9d59-b9abf9d41f64-kube-api-access-bcjnc\" (UniqueName: \"kubernetes.io/projected/924707d4-bd50-4ce6-9d59-b9abf9d41f64-kube-api-access-bcjnc\") pod \"924707d4-bd50-4ce6-9d59-b9abf9d41f64\" (UID: \"924707d4-bd50-4ce6-9d59-b9abf9d41f64\") "
Nov 03 00:17:16.592678 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.592661    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924707d4-bd50-4ce6-9d59-b9abf9d41f64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924707d4-bd50-4ce6-9d59-b9abf9d41f64-plugins-config-volume\") pod \"924707d4-bd50-4ce6-9d59-b9abf9d41f64\" (UID: \"924707d4-bd50-4ce6-9d59-b9abf9d41f64\") "
Nov 03 00:17:16.593069 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.593040    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924707d4-bd50-4ce6-9d59-b9abf9d41f64-plugins-config-volume" pod "924707d4-bd50-4ce6-9d59-b9abf9d41f64" (UID: "924707d4-bd50-4ce6-9d59-b9abf9d41f64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:17:16.593224 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.593201    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4drb\" (UniqueName: \"kubernetes.io/projected/959479c1-5aad-4927-9d79-384ef72ebf9e-kube-api-access-x4drb\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:16.593224 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.593225    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/959479c1-5aad-4927-9d79-384ef72ebf9e-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:16.593429 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.593310    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924707d4-bd50-4ce6-9d59-b9abf9d41f64-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:16.594694 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.594674    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924707d4-bd50-4ce6-9d59-b9abf9d41f64-kube-api-access-bcjnc" pod "924707d4-bd50-4ce6-9d59-b9abf9d41f64" (UID: "924707d4-bd50-4ce6-9d59-b9abf9d41f64"). InnerVolumeSpecName "kube-api-access-bcjnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:17:16.599341 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:17:16.599317    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7427-e089fb\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="3da850c4-ee0b-4a88-a421-7f8800ba06e2" pod="opendatahub/test-curl-7427-e089fb"
Nov 03 00:17:16.648041 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:17:16.648006    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7427-a3ac02\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="b86a51a3-351e-46c3-ad35-68eb39e6ee8d" pod="opendatahub/test-curl-7427-a3ac02"
Nov 03 00:17:16.694480 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.694449    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcjnc\" (UniqueName: \"kubernetes.io/projected/924707d4-bd50-4ce6-9d59-b9abf9d41f64-kube-api-access-bcjnc\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:16.724447 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:16.724414    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:17.047443 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:17.047406    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:17.404712 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:17.404682    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj"
Nov 03 00:17:17.404893 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:17.404681    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj" event={"ID":"924707d4-bd50-4ce6-9d59-b9abf9d41f64","Type":"ContainerDied","Data":"f2496f4ba02eaa5b4cc07b3e418fc87a7a6fdffe0de6864c2000f64bf515729b"}
Nov 03 00:17:17.404893 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:17.404825    4294 scope.go:122] "RemoveContainer" containerID="12aadbb0dc40ddd4e4ec2d9c212d1d7b8cb08a304c38c7735a4b2ab95ed7e8f4"
Nov 03 00:17:17.406519 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:17.406501    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7"
Nov 03 00:17:17.406623 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:17.406522    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7" event={"ID":"959479c1-5aad-4927-9d79-384ef72ebf9e","Type":"ContainerDied","Data":"47ac25150737d85c25137456ce4a7910192ea454f3e99823e0c6b8f5b9373885"}
Nov 03 00:17:17.415851 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:17.415822    4294 scope.go:122] "RemoveContainer" containerID="9d6bb2e39dd7b760a297d506b011c6395b8fe234a087751b4eb88f7bd21c3b49"
Nov 03 00:17:17.561101 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:17.561070    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7"]
Nov 03 00:17:17.567416 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:17.567388    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-ba1918-77598c9f6c-8qhv7"]
Nov 03 00:17:17.675364 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:17.675278    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj"]
Nov 03 00:17:17.677878 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:17.677855    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-ba1918-84f9c85c79-p5lhj"]
Nov 03 00:17:18.074791 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:18.074757    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:18.249407 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:18.249363    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:18.548303 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:18.548270    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="924707d4-bd50-4ce6-9d59-b9abf9d41f64" path="/var/lib/kubelet/pods/924707d4-bd50-4ce6-9d59-b9abf9d41f64/volumes"
Nov 03 00:17:18.548671 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:18.548659    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="959479c1-5aad-4927-9d79-384ef72ebf9e" path="/var/lib/kubelet/pods/959479c1-5aad-4927-9d79-384ef72ebf9e/volumes"
Nov 03 00:17:18.645708 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:18.645685    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7427-a3ac02"
Nov 03 00:17:18.715795 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:18.715710    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86a51a3-351e-46c3-ad35-68eb39e6ee8d-kube-api-access-7n4gg\" (UniqueName: \"kubernetes.io/projected/b86a51a3-351e-46c3-ad35-68eb39e6ee8d-kube-api-access-7n4gg\") pod \"b86a51a3-351e-46c3-ad35-68eb39e6ee8d\" (UID: \"b86a51a3-351e-46c3-ad35-68eb39e6ee8d\") "
Nov 03 00:17:18.717893 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:18.717865    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86a51a3-351e-46c3-ad35-68eb39e6ee8d-kube-api-access-7n4gg" pod "b86a51a3-351e-46c3-ad35-68eb39e6ee8d" (UID: "b86a51a3-351e-46c3-ad35-68eb39e6ee8d"). InnerVolumeSpecName "kube-api-access-7n4gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:17:18.816970 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:18.816936    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n4gg\" (UniqueName: \"kubernetes.io/projected/b86a51a3-351e-46c3-ad35-68eb39e6ee8d-kube-api-access-7n4gg\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:19.421199 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.421162    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48" podUID="785b24ed-8d15-410f-b7d4-02549e6e2201" containerName="main" containerID="cri-o://a814ac8af58629c4a3c2c4002498978b5ca801ee330db383be5afe2c5fd8ffe0" gracePeriod=45
Nov 03 00:17:19.423557 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.423531    4294 generic.go:356] "Generic (PLEG): container finished" podID="b86a51a3-351e-46c3-ad35-68eb39e6ee8d" containerID="b31d05b7637f259809114f1bb6cd3971e61b84b08c3150ce11b73e871bd1100f" exitCode=137
Nov 03 00:17:19.423690 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.423574    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7427-a3ac02"
Nov 03 00:17:19.423690 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.423640    4294 scope.go:122] "RemoveContainer" containerID="b31d05b7637f259809114f1bb6cd3971e61b84b08c3150ce11b73e871bd1100f"
Nov 03 00:17:19.434229 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.434203    4294 scope.go:122] "RemoveContainer" containerID="b31d05b7637f259809114f1bb6cd3971e61b84b08c3150ce11b73e871bd1100f"
Nov 03 00:17:19.434522 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:17:19.434503    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b31d05b7637f259809114f1bb6cd3971e61b84b08c3150ce11b73e871bd1100f\": container with ID starting with b31d05b7637f259809114f1bb6cd3971e61b84b08c3150ce11b73e871bd1100f not found: ID does not exist" containerID="b31d05b7637f259809114f1bb6cd3971e61b84b08c3150ce11b73e871bd1100f"
Nov 03 00:17:19.434576 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.434531    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b31d05b7637f259809114f1bb6cd3971e61b84b08c3150ce11b73e871bd1100f"} err="failed to get container status \"b31d05b7637f259809114f1bb6cd3971e61b84b08c3150ce11b73e871bd1100f\": rpc error: code = NotFound desc = could not find container \"b31d05b7637f259809114f1bb6cd3971e61b84b08c3150ce11b73e871bd1100f\": container with ID starting with b31d05b7637f259809114f1bb6cd3971e61b84b08c3150ce11b73e871bd1100f not found: ID does not exist"
Nov 03 00:17:19.509957 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.509922    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48" podUID="785b24ed-8d15-410f-b7d4-02549e6e2201" containerName="main" probeResult="failure" output="Get \"https://10.133.0.204:8000/ready\": dial tcp 10.133.0.204:8000: connect: connection refused"
Nov 03 00:17:19.535235 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:17:19.535202    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7427-a3ac02\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="b86a51a3-351e-46c3-ad35-68eb39e6ee8d" pod="opendatahub/test-curl-7427-a3ac02"
Nov 03 00:17:19.543944 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.543915    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:19.683947 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.683893    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:17:19.732807 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:17:19.732769    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7427-a3ac02\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="b86a51a3-351e-46c3-ad35-68eb39e6ee8d" pod="opendatahub/test-curl-7427-a3ac02"
Nov 03 00:17:19.827451 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.827417    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-home\" (UniqueName: \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-home\") pod \"785b24ed-8d15-410f-b7d4-02549e6e2201\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") "
Nov 03 00:17:19.827651 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.827498    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-model-cache\") pod \"785b24ed-8d15-410f-b7d4-02549e6e2201\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") "
Nov 03 00:17:19.827651 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.827570    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/785b24ed-8d15-410f-b7d4-02549e6e2201-tls-certs\" (UniqueName: \"kubernetes.io/secret/785b24ed-8d15-410f-b7d4-02549e6e2201-tls-certs\") pod \"785b24ed-8d15-410f-b7d4-02549e6e2201\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") "
Nov 03 00:17:19.827651 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.827603    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-tmp-dir\") pod \"785b24ed-8d15-410f-b7d4-02549e6e2201\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") "
Nov 03 00:17:19.827651 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.827625    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-dshm\" (UniqueName: \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-dshm\") pod \"785b24ed-8d15-410f-b7d4-02549e6e2201\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") "
Nov 03 00:17:19.827866 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.827681    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-home" pod "785b24ed-8d15-410f-b7d4-02549e6e2201" (UID: "785b24ed-8d15-410f-b7d4-02549e6e2201"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:17:19.827866 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.827704    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/785b24ed-8d15-410f-b7d4-02549e6e2201-kube-api-access-s575q\" (UniqueName: \"kubernetes.io/projected/785b24ed-8d15-410f-b7d4-02549e6e2201-kube-api-access-s575q\") pod \"785b24ed-8d15-410f-b7d4-02549e6e2201\" (UID: \"785b24ed-8d15-410f-b7d4-02549e6e2201\") "
Nov 03 00:17:19.827866 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.827731    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-model-cache" pod "785b24ed-8d15-410f-b7d4-02549e6e2201" (UID: "785b24ed-8d15-410f-b7d4-02549e6e2201"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:17:19.828020 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.827882    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-tmp-dir" pod "785b24ed-8d15-410f-b7d4-02549e6e2201" (UID: "785b24ed-8d15-410f-b7d4-02549e6e2201"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:17:19.828157 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.828132    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:19.828157 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.828156    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:19.828320 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.828171    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:19.829777 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.829757    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-dshm" pod "785b24ed-8d15-410f-b7d4-02549e6e2201" (UID: "785b24ed-8d15-410f-b7d4-02549e6e2201"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:17:19.829860 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.829801    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/785b24ed-8d15-410f-b7d4-02549e6e2201-tls-certs" pod "785b24ed-8d15-410f-b7d4-02549e6e2201" (UID: "785b24ed-8d15-410f-b7d4-02549e6e2201"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:17:19.829932 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.829913    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/785b24ed-8d15-410f-b7d4-02549e6e2201-kube-api-access-s575q" pod "785b24ed-8d15-410f-b7d4-02549e6e2201" (UID: "785b24ed-8d15-410f-b7d4-02549e6e2201"). InnerVolumeSpecName "kube-api-access-s575q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:17:19.929118 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.929078    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/785b24ed-8d15-410f-b7d4-02549e6e2201-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:19.929118 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.929108    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/785b24ed-8d15-410f-b7d4-02549e6e2201-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:19.929118 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:19.929124    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s575q\" (UniqueName: \"kubernetes.io/projected/785b24ed-8d15-410f-b7d4-02549e6e2201-kube-api-access-s575q\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:20.059025 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:20.058969    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:20.436149 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:20.436061    4294 generic.go:356] "Generic (PLEG): container finished" podID="785b24ed-8d15-410f-b7d4-02549e6e2201" containerID="a814ac8af58629c4a3c2c4002498978b5ca801ee330db383be5afe2c5fd8ffe0" exitCode=0
Nov 03 00:17:20.436149 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:20.436102    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"
Nov 03 00:17:20.436658 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:20.436145    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48" event={"ID":"785b24ed-8d15-410f-b7d4-02549e6e2201","Type":"ContainerDied","Data":"a814ac8af58629c4a3c2c4002498978b5ca801ee330db383be5afe2c5fd8ffe0"}
Nov 03 00:17:20.436658 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:20.436190    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48" event={"ID":"785b24ed-8d15-410f-b7d4-02549e6e2201","Type":"ContainerDied","Data":"a8f41df67a993e3f53e8f6c22863a50cff6b68d8d1a691ca225aab6ad29bf4d1"}
Nov 03 00:17:20.436658 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:20.436209    4294 scope.go:122] "RemoveContainer" containerID="a814ac8af58629c4a3c2c4002498978b5ca801ee330db383be5afe2c5fd8ffe0"
Nov 03 00:17:20.447363 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:20.447337    4294 scope.go:122] "RemoveContainer" containerID="a814ac8af58629c4a3c2c4002498978b5ca801ee330db383be5afe2c5fd8ffe0"
Nov 03 00:17:20.447656 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:17:20.447638    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a814ac8af58629c4a3c2c4002498978b5ca801ee330db383be5afe2c5fd8ffe0\": container with ID starting with a814ac8af58629c4a3c2c4002498978b5ca801ee330db383be5afe2c5fd8ffe0 not found: ID does not exist" containerID="a814ac8af58629c4a3c2c4002498978b5ca801ee330db383be5afe2c5fd8ffe0"
Nov 03 00:17:20.447724 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:20.447669    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a814ac8af58629c4a3c2c4002498978b5ca801ee330db383be5afe2c5fd8ffe0"} err="failed to get container status \"a814ac8af58629c4a3c2c4002498978b5ca801ee330db383be5afe2c5fd8ffe0\": rpc error: code = NotFound desc = could not find container \"a814ac8af58629c4a3c2c4002498978b5ca801ee330db383be5afe2c5fd8ffe0\": container with ID starting with a814ac8af58629c4a3c2c4002498978b5ca801ee330db383be5afe2c5fd8ffe0 not found: ID does not exist"
Nov 03 00:17:20.484955 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:17:20.484916    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7427-a3ac02\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="b86a51a3-351e-46c3-ad35-68eb39e6ee8d" pod="opendatahub/test-curl-7427-a3ac02"
Nov 03 00:17:20.547262 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:20.547234    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b86a51a3-351e-46c3-ad35-68eb39e6ee8d" path="/var/lib/kubelet/pods/b86a51a3-351e-46c3-ad35-68eb39e6ee8d/volumes"
Nov 03 00:17:20.584986 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:17:20.584950    4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7427-a3ac02\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-2.compute.internal' and this object" podUID="b86a51a3-351e-46c3-ad35-68eb39e6ee8d" pod="opendatahub/test-curl-7427-a3ac02"
Nov 03 00:17:20.692175 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:20.692090    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"]
Nov 03 00:17:20.739849 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:20.739803    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7ca60f/sub-test-model-7ca60f-kserve-6bdb8657dc-xjc48"]
Nov 03 00:17:20.752552 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:20.752521    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:21.597052 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:21.597020    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:22.439382 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:22.439336    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76" podUID="95ad5c99-119f-46ca-8664-8f2878dbca94" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:17:22.547592 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:22.547557    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="785b24ed-8d15-410f-b7d4-02549e6e2201" path="/var/lib/kubelet/pods/785b24ed-8d15-410f-b7d4-02549e6e2201/volumes"
Nov 03 00:17:23.212463 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:23.212429    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:23.446094 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:23.446057    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq" podUID="36cb1d31-8467-441f-90bf-07ba20b0f3c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:17:24.004568 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:24.004477    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f" podUID="b716a68d-1d7e-48d6-98a5-6fdc5472cac9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:17:25.009480 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:25.009440    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh" podUID="78e187a5-e905-410b-a2c5-ddced7693786" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:17:26.588854 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:26.588809    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:27.196276 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:27.196241    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:27.318304 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:27.318266    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj"]
Nov 03 00:17:27.369877 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:27.369839    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:27.616146 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:27.616112    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:27.802600 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:27.802566    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:28.260245 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:28.260203    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:28.990333 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:28.990293    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:30.694315 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:30.694276    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"]
Nov 03 00:17:30.694838 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:30.694526    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn" podUID="a2210e3e-ee40-4277-8865-93acdf8e81a4" containerName="maas-api" containerID="cri-o://1e7984386ece16c38a92cdd927b5cbb0968c427e76e94ddd9626c4f3b1f39654" gracePeriod=30
Nov 03 00:17:30.867955 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:30.867865    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:31.020888 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.020858    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"
Nov 03 00:17:31.152097 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.152055    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2210e3e-ee40-4277-8865-93acdf8e81a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2210e3e-ee40-4277-8865-93acdf8e81a4-openshift-service-ca\") pod \"a2210e3e-ee40-4277-8865-93acdf8e81a4\" (UID: \"a2210e3e-ee40-4277-8865-93acdf8e81a4\") "
Nov 03 00:17:31.152307 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.152121    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2210e3e-ee40-4277-8865-93acdf8e81a4-kube-api-access-4g9k2\" (UniqueName: \"kubernetes.io/projected/a2210e3e-ee40-4277-8865-93acdf8e81a4-kube-api-access-4g9k2\") pod \"a2210e3e-ee40-4277-8865-93acdf8e81a4\" (UID: \"a2210e3e-ee40-4277-8865-93acdf8e81a4\") "
Nov 03 00:17:31.152307 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.152277    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2210e3e-ee40-4277-8865-93acdf8e81a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2210e3e-ee40-4277-8865-93acdf8e81a4-maas-api-tls\") pod \"a2210e3e-ee40-4277-8865-93acdf8e81a4\" (UID: \"a2210e3e-ee40-4277-8865-93acdf8e81a4\") "
Nov 03 00:17:31.152458 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.152339    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2210e3e-ee40-4277-8865-93acdf8e81a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2210e3e-ee40-4277-8865-93acdf8e81a4-metrics-certs\") pod \"a2210e3e-ee40-4277-8865-93acdf8e81a4\" (UID: \"a2210e3e-ee40-4277-8865-93acdf8e81a4\") "
Nov 03 00:17:31.152524 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.152499    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2210e3e-ee40-4277-8865-93acdf8e81a4-openshift-service-ca" pod "a2210e3e-ee40-4277-8865-93acdf8e81a4" (UID: "a2210e3e-ee40-4277-8865-93acdf8e81a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:17:31.152811 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.152783    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2210e3e-ee40-4277-8865-93acdf8e81a4-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:31.154518 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.154484    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2210e3e-ee40-4277-8865-93acdf8e81a4-metrics-certs" pod "a2210e3e-ee40-4277-8865-93acdf8e81a4" (UID: "a2210e3e-ee40-4277-8865-93acdf8e81a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:17:31.154518 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.154504    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2210e3e-ee40-4277-8865-93acdf8e81a4-maas-api-tls" pod "a2210e3e-ee40-4277-8865-93acdf8e81a4" (UID: "a2210e3e-ee40-4277-8865-93acdf8e81a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:17:31.154761 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.154519    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2210e3e-ee40-4277-8865-93acdf8e81a4-kube-api-access-4g9k2" pod "a2210e3e-ee40-4277-8865-93acdf8e81a4" (UID: "a2210e3e-ee40-4277-8865-93acdf8e81a4"). InnerVolumeSpecName "kube-api-access-4g9k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:17:31.253913 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.253831    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2210e3e-ee40-4277-8865-93acdf8e81a4-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:31.253913 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.253862    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g9k2\" (UniqueName: \"kubernetes.io/projected/a2210e3e-ee40-4277-8865-93acdf8e81a4-kube-api-access-4g9k2\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:31.253913 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.253873    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2210e3e-ee40-4277-8865-93acdf8e81a4-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:31.504557 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.504470    4294 generic.go:356] "Generic (PLEG): container finished" podID="a2210e3e-ee40-4277-8865-93acdf8e81a4" containerID="1e7984386ece16c38a92cdd927b5cbb0968c427e76e94ddd9626c4f3b1f39654" exitCode=0
Nov 03 00:17:31.504557 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.504523    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"
Nov 03 00:17:31.504764 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.504557    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn" event={"ID":"a2210e3e-ee40-4277-8865-93acdf8e81a4","Type":"ContainerDied","Data":"1e7984386ece16c38a92cdd927b5cbb0968c427e76e94ddd9626c4f3b1f39654"}
Nov 03 00:17:31.504764 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.504616    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn" event={"ID":"a2210e3e-ee40-4277-8865-93acdf8e81a4","Type":"ContainerDied","Data":"a5b6291af0ce59ba45615b60cd29e7b65faec9e237ffbd27ffd0b5f5d5334d57"}
Nov 03 00:17:31.504764 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.504642    4294 scope.go:122] "RemoveContainer" containerID="1e7984386ece16c38a92cdd927b5cbb0968c427e76e94ddd9626c4f3b1f39654"
Nov 03 00:17:31.516729 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.516707    4294 scope.go:122] "RemoveContainer" containerID="1e7984386ece16c38a92cdd927b5cbb0968c427e76e94ddd9626c4f3b1f39654"
Nov 03 00:17:31.517121 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:17:31.517093    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e7984386ece16c38a92cdd927b5cbb0968c427e76e94ddd9626c4f3b1f39654\": container with ID starting with 1e7984386ece16c38a92cdd927b5cbb0968c427e76e94ddd9626c4f3b1f39654 not found: ID does not exist" containerID="1e7984386ece16c38a92cdd927b5cbb0968c427e76e94ddd9626c4f3b1f39654"
Nov 03 00:17:31.517206 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.517130    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e7984386ece16c38a92cdd927b5cbb0968c427e76e94ddd9626c4f3b1f39654"} err="failed to get container status \"1e7984386ece16c38a92cdd927b5cbb0968c427e76e94ddd9626c4f3b1f39654\": rpc error: code = NotFound desc = could not find container \"1e7984386ece16c38a92cdd927b5cbb0968c427e76e94ddd9626c4f3b1f39654\": container with ID starting with 1e7984386ece16c38a92cdd927b5cbb0968c427e76e94ddd9626c4f3b1f39654 not found: ID does not exist"
Nov 03 00:17:31.664608 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.664572    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"]
Nov 03 00:17:31.710942 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.710905    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3d8404-79b69dfcdd-f96hn"]
Nov 03 00:17:31.875222 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.874849    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh"
Nov 03 00:17:31.940582 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.940551    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:31.965012 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.964950    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e187a5-e905-410b-a2c5-ddced7693786-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e187a5-e905-410b-a2c5-ddced7693786-plugins-config-volume\") pod \"78e187a5-e905-410b-a2c5-ddced7693786\" (UID: \"78e187a5-e905-410b-a2c5-ddced7693786\") "
Nov 03 00:17:31.965168 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.965067    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e187a5-e905-410b-a2c5-ddced7693786-kube-api-access-f8p7p\" (UniqueName: \"kubernetes.io/projected/78e187a5-e905-410b-a2c5-ddced7693786-kube-api-access-f8p7p\") pod \"78e187a5-e905-410b-a2c5-ddced7693786\" (UID: \"78e187a5-e905-410b-a2c5-ddced7693786\") "
Nov 03 00:17:31.965347 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.965326    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e187a5-e905-410b-a2c5-ddced7693786-plugins-config-volume" pod "78e187a5-e905-410b-a2c5-ddced7693786" (UID: "78e187a5-e905-410b-a2c5-ddced7693786"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:17:31.967241 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.967211    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e187a5-e905-410b-a2c5-ddced7693786-kube-api-access-f8p7p" pod "78e187a5-e905-410b-a2c5-ddced7693786" (UID: "78e187a5-e905-410b-a2c5-ddced7693786"). InnerVolumeSpecName "kube-api-access-f8p7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:17:31.977906 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:31.977878    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f"
Nov 03 00:17:32.067083 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.066990    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8p7p\" (UniqueName: \"kubernetes.io/projected/78e187a5-e905-410b-a2c5-ddced7693786-kube-api-access-f8p7p\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:32.067083 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.067025    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e187a5-e905-410b-a2c5-ddced7693786-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:32.168400 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.168360    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b716a68d-1d7e-48d6-98a5-6fdc5472cac9-kube-api-access-k7m7p\" (UniqueName: \"kubernetes.io/projected/b716a68d-1d7e-48d6-98a5-6fdc5472cac9-kube-api-access-k7m7p\") pod \"b716a68d-1d7e-48d6-98a5-6fdc5472cac9\" (UID: \"b716a68d-1d7e-48d6-98a5-6fdc5472cac9\") "
Nov 03 00:17:32.168400 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.168402    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b716a68d-1d7e-48d6-98a5-6fdc5472cac9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b716a68d-1d7e-48d6-98a5-6fdc5472cac9-plugins-config-volume\") pod \"b716a68d-1d7e-48d6-98a5-6fdc5472cac9\" (UID: \"b716a68d-1d7e-48d6-98a5-6fdc5472cac9\") "
Nov 03 00:17:32.168859 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.168832    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b716a68d-1d7e-48d6-98a5-6fdc5472cac9-plugins-config-volume" pod "b716a68d-1d7e-48d6-98a5-6fdc5472cac9" (UID: "b716a68d-1d7e-48d6-98a5-6fdc5472cac9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:17:32.169042 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.169026    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b716a68d-1d7e-48d6-98a5-6fdc5472cac9-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:32.170593 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.170574    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b716a68d-1d7e-48d6-98a5-6fdc5472cac9-kube-api-access-k7m7p" pod "b716a68d-1d7e-48d6-98a5-6fdc5472cac9" (UID: "b716a68d-1d7e-48d6-98a5-6fdc5472cac9"). InnerVolumeSpecName "kube-api-access-k7m7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:17:32.269905 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.269836    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7m7p\" (UniqueName: \"kubernetes.io/projected/b716a68d-1d7e-48d6-98a5-6fdc5472cac9-kube-api-access-k7m7p\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:32.319357 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.319218    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj" podUID="edca56c1-37b2-4055-bb5a-d43f3d6cb47a" containerName="payload-pre-processing" containerID="cri-o://6f1b86e19226c1434e57926cf7a2538930933c6fae8775196e862ba76705a352" gracePeriod=25
Nov 03 00:17:32.439916 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.439878    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76" podUID="95ad5c99-119f-46ca-8664-8f2878dbca94" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:17:32.513199 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.513165    4294 generic.go:356] "Generic (PLEG): container finished" podID="b716a68d-1d7e-48d6-98a5-6fdc5472cac9" containerID="c422267b001cf69d1eae9ddb63f2283ee6bbd619e80870d35512f03dc0418783" exitCode=0
Nov 03 00:17:32.513406 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.513255    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f"
Nov 03 00:17:32.513406 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.513298    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f" event={"ID":"b716a68d-1d7e-48d6-98a5-6fdc5472cac9","Type":"ContainerDied","Data":"c422267b001cf69d1eae9ddb63f2283ee6bbd619e80870d35512f03dc0418783"}
Nov 03 00:17:32.513406 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.513357    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f" event={"ID":"b716a68d-1d7e-48d6-98a5-6fdc5472cac9","Type":"ContainerDied","Data":"b394a349200e09e765c04c3939d33dc440e8378c7f05b1030800f4fbec9f2a9a"}
Nov 03 00:17:32.513406 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.513393    4294 scope.go:122] "RemoveContainer" containerID="c422267b001cf69d1eae9ddb63f2283ee6bbd619e80870d35512f03dc0418783"
Nov 03 00:17:32.515006 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.514937    4294 generic.go:356] "Generic (PLEG): container finished" podID="78e187a5-e905-410b-a2c5-ddced7693786" containerID="de513cd3a4405ddf97a023f3ed2226e1536e7cef74f34cb24b5527d5b6ec96c6" exitCode=0
Nov 03 00:17:32.515132 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.515106    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh"
Nov 03 00:17:32.517747 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.515774    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh" event={"ID":"78e187a5-e905-410b-a2c5-ddced7693786","Type":"ContainerDied","Data":"de513cd3a4405ddf97a023f3ed2226e1536e7cef74f34cb24b5527d5b6ec96c6"}
Nov 03 00:17:32.517747 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.515856    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh" event={"ID":"78e187a5-e905-410b-a2c5-ddced7693786","Type":"ContainerDied","Data":"3bc2ae83186ed2abe827ad8af4356f01c52a2cd778dcfdef7992b8dbd643b999"}
Nov 03 00:17:32.533888 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.533864    4294 scope.go:122] "RemoveContainer" containerID="c422267b001cf69d1eae9ddb63f2283ee6bbd619e80870d35512f03dc0418783"
Nov 03 00:17:32.534328 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:17:32.534297    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c422267b001cf69d1eae9ddb63f2283ee6bbd619e80870d35512f03dc0418783\": container with ID starting with c422267b001cf69d1eae9ddb63f2283ee6bbd619e80870d35512f03dc0418783 not found: ID does not exist" containerID="c422267b001cf69d1eae9ddb63f2283ee6bbd619e80870d35512f03dc0418783"
Nov 03 00:17:32.534461 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.534333    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c422267b001cf69d1eae9ddb63f2283ee6bbd619e80870d35512f03dc0418783"} err="failed to get container status \"c422267b001cf69d1eae9ddb63f2283ee6bbd619e80870d35512f03dc0418783\": rpc error: code = NotFound desc = could not find container \"c422267b001cf69d1eae9ddb63f2283ee6bbd619e80870d35512f03dc0418783\": container with ID starting with c422267b001cf69d1eae9ddb63f2283ee6bbd619e80870d35512f03dc0418783 not found: ID does not exist"
Nov 03 00:17:32.534461 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.534351    4294 scope.go:122] "RemoveContainer" containerID="de513cd3a4405ddf97a023f3ed2226e1536e7cef74f34cb24b5527d5b6ec96c6"
Nov 03 00:17:32.545783 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.545692    4294 scope.go:122] "RemoveContainer" containerID="de513cd3a4405ddf97a023f3ed2226e1536e7cef74f34cb24b5527d5b6ec96c6"
Nov 03 00:17:32.546028 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:17:32.545995    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de513cd3a4405ddf97a023f3ed2226e1536e7cef74f34cb24b5527d5b6ec96c6\": container with ID starting with de513cd3a4405ddf97a023f3ed2226e1536e7cef74f34cb24b5527d5b6ec96c6 not found: ID does not exist" containerID="de513cd3a4405ddf97a023f3ed2226e1536e7cef74f34cb24b5527d5b6ec96c6"
Nov 03 00:17:32.546161 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.546037    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de513cd3a4405ddf97a023f3ed2226e1536e7cef74f34cb24b5527d5b6ec96c6"} err="failed to get container status \"de513cd3a4405ddf97a023f3ed2226e1536e7cef74f34cb24b5527d5b6ec96c6\": rpc error: code = NotFound desc = could not find container \"de513cd3a4405ddf97a023f3ed2226e1536e7cef74f34cb24b5527d5b6ec96c6\": container with ID starting with de513cd3a4405ddf97a023f3ed2226e1536e7cef74f34cb24b5527d5b6ec96c6 not found: ID does not exist"
Nov 03 00:17:32.547386 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.547361    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2210e3e-ee40-4277-8865-93acdf8e81a4" path="/var/lib/kubelet/pods/a2210e3e-ee40-4277-8865-93acdf8e81a4/volumes"
Nov 03 00:17:32.669936 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.669845    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh"]
Nov 03 00:17:32.674506 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.674473    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-3d8404-7b54b9765f-24njh"]
Nov 03 00:17:32.780135 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.780096    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f"]
Nov 03 00:17:32.787989 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:32.787949    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-3d8404-6556f984f5-bn44f"]
Nov 03 00:17:33.387934 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:33.387319    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:33.457149 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:33.457102    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq" podUID="36cb1d31-8467-441f-90bf-07ba20b0f3c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:17:33.600648 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:33.600603    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:34.016352 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:34.016312    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:34.548483 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:34.548448    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e187a5-e905-410b-a2c5-ddced7693786" path="/var/lib/kubelet/pods/78e187a5-e905-410b-a2c5-ddced7693786/volumes"
Nov 03 00:17:34.548856 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:34.548842    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b716a68d-1d7e-48d6-98a5-6fdc5472cac9" path="/var/lib/kubelet/pods/b716a68d-1d7e-48d6-98a5-6fdc5472cac9/volumes"
Nov 03 00:17:34.890630 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:34.890510    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:35.357388 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:35.357353    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"]
Nov 03 00:17:35.357895 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:35.357580    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29" podUID="da283988-fca2-4bde-8256-4589d9536e4b" containerName="istio-proxy" containerID="cri-o://54cb2277ba4e27b198d663aba7731387d3efbb212569dea636b45c3eda572a06" gracePeriod=30
Nov 03 00:17:35.411534 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:35.411441    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:35.800555 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:35.800512    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:35.997060 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:35.997021    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:36.315641 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.315556    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq"
Nov 03 00:17:36.322707 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.322667    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36cb1d31-8467-441f-90bf-07ba20b0f3c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36cb1d31-8467-441f-90bf-07ba20b0f3c6-plugins-config-volume\") pod \"36cb1d31-8467-441f-90bf-07ba20b0f3c6\" (UID: \"36cb1d31-8467-441f-90bf-07ba20b0f3c6\") "
Nov 03 00:17:36.322908 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.322735    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36cb1d31-8467-441f-90bf-07ba20b0f3c6-kube-api-access-hnvn2\" (UniqueName: \"kubernetes.io/projected/36cb1d31-8467-441f-90bf-07ba20b0f3c6-kube-api-access-hnvn2\") pod \"36cb1d31-8467-441f-90bf-07ba20b0f3c6\" (UID: \"36cb1d31-8467-441f-90bf-07ba20b0f3c6\") "
Nov 03 00:17:36.323139 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.323109    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36cb1d31-8467-441f-90bf-07ba20b0f3c6-plugins-config-volume" pod "36cb1d31-8467-441f-90bf-07ba20b0f3c6" (UID: "36cb1d31-8467-441f-90bf-07ba20b0f3c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:17:36.325650 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.325582    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36cb1d31-8467-441f-90bf-07ba20b0f3c6-kube-api-access-hnvn2" pod "36cb1d31-8467-441f-90bf-07ba20b0f3c6" (UID: "36cb1d31-8467-441f-90bf-07ba20b0f3c6"). InnerVolumeSpecName "kube-api-access-hnvn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:17:36.397397 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.397367    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76"
Nov 03 00:17:36.423528 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.423495    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95ad5c99-119f-46ca-8664-8f2878dbca94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95ad5c99-119f-46ca-8664-8f2878dbca94-plugins-config-volume\") pod \"95ad5c99-119f-46ca-8664-8f2878dbca94\" (UID: \"95ad5c99-119f-46ca-8664-8f2878dbca94\") "
Nov 03 00:17:36.423708 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.423540    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ad5c99-119f-46ca-8664-8f2878dbca94-kube-api-access-dcxqj\" (UniqueName: \"kubernetes.io/projected/95ad5c99-119f-46ca-8664-8f2878dbca94-kube-api-access-dcxqj\") pod \"95ad5c99-119f-46ca-8664-8f2878dbca94\" (UID: \"95ad5c99-119f-46ca-8664-8f2878dbca94\") "
Nov 03 00:17:36.423831 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.423818    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36cb1d31-8467-441f-90bf-07ba20b0f3c6-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:36.423913 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.423834    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnvn2\" (UniqueName: \"kubernetes.io/projected/36cb1d31-8467-441f-90bf-07ba20b0f3c6-kube-api-access-hnvn2\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:36.424037 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.424010    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95ad5c99-119f-46ca-8664-8f2878dbca94-plugins-config-volume" pod "95ad5c99-119f-46ca-8664-8f2878dbca94" (UID: "95ad5c99-119f-46ca-8664-8f2878dbca94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:17:36.426264 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.426235    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ad5c99-119f-46ca-8664-8f2878dbca94-kube-api-access-dcxqj" pod "95ad5c99-119f-46ca-8664-8f2878dbca94" (UID: "95ad5c99-119f-46ca-8664-8f2878dbca94"). InnerVolumeSpecName "kube-api-access-dcxqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:17:36.525164 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.525121    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95ad5c99-119f-46ca-8664-8f2878dbca94-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:36.525164 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.525156    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcxqj\" (UniqueName: \"kubernetes.io/projected/95ad5c99-119f-46ca-8664-8f2878dbca94-kube-api-access-dcxqj\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:36.576091 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.575968    4294 generic.go:356] "Generic (PLEG): container finished" podID="95ad5c99-119f-46ca-8664-8f2878dbca94" containerID="f18166f5e45be864f02b4bf751c71292a3084367dcb9f04b632d389284f77dea" exitCode=0
Nov 03 00:17:36.576091 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.576028    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76"
Nov 03 00:17:36.576091 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.576012    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76" event={"ID":"95ad5c99-119f-46ca-8664-8f2878dbca94","Type":"ContainerDied","Data":"f18166f5e45be864f02b4bf751c71292a3084367dcb9f04b632d389284f77dea"}
Nov 03 00:17:36.576390 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.576128    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76" event={"ID":"95ad5c99-119f-46ca-8664-8f2878dbca94","Type":"ContainerDied","Data":"c0f1deb273b9fe1c7f964635cea9af49513ec50621d1b731a3ee0eac3c880be4"}
Nov 03 00:17:36.576390 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.576155    4294 scope.go:122] "RemoveContainer" containerID="f18166f5e45be864f02b4bf751c71292a3084367dcb9f04b632d389284f77dea"
Nov 03 00:17:36.577661 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.577636    4294 generic.go:356] "Generic (PLEG): container finished" podID="36cb1d31-8467-441f-90bf-07ba20b0f3c6" containerID="a2f19b1757ca0654957594e93f52980ad5d3a0302b8034a247d47c984c66ac93" exitCode=0
Nov 03 00:17:36.577778 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.577677    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq"
Nov 03 00:17:36.577778 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.577705    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq" event={"ID":"36cb1d31-8467-441f-90bf-07ba20b0f3c6","Type":"ContainerDied","Data":"a2f19b1757ca0654957594e93f52980ad5d3a0302b8034a247d47c984c66ac93"}
Nov 03 00:17:36.577778 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.577748    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq" event={"ID":"36cb1d31-8467-441f-90bf-07ba20b0f3c6","Type":"ContainerDied","Data":"f3a196d7f8458d5d331253459563c454ce5d03f5705dfb888f0dae9bae065699"}
Nov 03 00:17:36.585908 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.585883    4294 scope.go:122] "RemoveContainer" containerID="f18166f5e45be864f02b4bf751c71292a3084367dcb9f04b632d389284f77dea"
Nov 03 00:17:36.586288 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:17:36.586263    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f18166f5e45be864f02b4bf751c71292a3084367dcb9f04b632d389284f77dea\": container with ID starting with f18166f5e45be864f02b4bf751c71292a3084367dcb9f04b632d389284f77dea not found: ID does not exist" containerID="f18166f5e45be864f02b4bf751c71292a3084367dcb9f04b632d389284f77dea"
Nov 03 00:17:36.586382 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.586297    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f18166f5e45be864f02b4bf751c71292a3084367dcb9f04b632d389284f77dea"} err="failed to get container status \"f18166f5e45be864f02b4bf751c71292a3084367dcb9f04b632d389284f77dea\": rpc error: code = NotFound desc = could not find container \"f18166f5e45be864f02b4bf751c71292a3084367dcb9f04b632d389284f77dea\": container with ID starting with f18166f5e45be864f02b4bf751c71292a3084367dcb9f04b632d389284f77dea not found: ID does not exist"
Nov 03 00:17:36.586382 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.586314    4294 scope.go:122] "RemoveContainer" containerID="a2f19b1757ca0654957594e93f52980ad5d3a0302b8034a247d47c984c66ac93"
Nov 03 00:17:36.595871 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.595850    4294 scope.go:122] "RemoveContainer" containerID="a2f19b1757ca0654957594e93f52980ad5d3a0302b8034a247d47c984c66ac93"
Nov 03 00:17:36.596216 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:17:36.596188    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f19b1757ca0654957594e93f52980ad5d3a0302b8034a247d47c984c66ac93\": container with ID starting with a2f19b1757ca0654957594e93f52980ad5d3a0302b8034a247d47c984c66ac93 not found: ID does not exist" containerID="a2f19b1757ca0654957594e93f52980ad5d3a0302b8034a247d47c984c66ac93"
Nov 03 00:17:36.596329 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.596222    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f19b1757ca0654957594e93f52980ad5d3a0302b8034a247d47c984c66ac93"} err="failed to get container status \"a2f19b1757ca0654957594e93f52980ad5d3a0302b8034a247d47c984c66ac93\": rpc error: code = NotFound desc = could not find container \"a2f19b1757ca0654957594e93f52980ad5d3a0302b8034a247d47c984c66ac93\": container with ID starting with a2f19b1757ca0654957594e93f52980ad5d3a0302b8034a247d47c984c66ac93 not found: ID does not exist"
Nov 03 00:17:36.732666 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.732627    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq"]
Nov 03 00:17:36.738361 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.738332    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-28f82c-6c9886d8b5-667fq"]
Nov 03 00:17:36.842797 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.842694    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76"]
Nov 03 00:17:36.846345 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:36.846309    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-28f82c-64949c574c-j5j76"]
Nov 03 00:17:37.290188 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:37.290158    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:37.453046 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:37.453007    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:37.803493 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:37.803452    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:37.958943 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:37.958910    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:38.547097 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:38.547065    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36cb1d31-8467-441f-90bf-07ba20b0f3c6" path="/var/lib/kubelet/pods/36cb1d31-8467-441f-90bf-07ba20b0f3c6/volumes"
Nov 03 00:17:38.547500 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:38.547441    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95ad5c99-119f-46ca-8664-8f2878dbca94" path="/var/lib/kubelet/pods/95ad5c99-119f-46ca-8664-8f2878dbca94/volumes"
Nov 03 00:17:38.695852 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:38.695819    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:38.894177 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:38.894093    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:39.164857 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:39.164767    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:40.273452 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.273416    4294 patch_prober.go:29] interesting pod/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:17:40.273856 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.273475    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29" podUID="da283988-fca2-4bde-8256-4589d9536e4b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:17:40.605559 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.605526    4294 generic.go:356] "Generic (PLEG): container finished" podID="da283988-fca2-4bde-8256-4589d9536e4b" containerID="54cb2277ba4e27b198d663aba7731387d3efbb212569dea636b45c3eda572a06" exitCode=0
Nov 03 00:17:40.605722 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.605605    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29" event={"ID":"da283988-fca2-4bde-8256-4589d9536e4b","Type":"ContainerDied","Data":"54cb2277ba4e27b198d663aba7731387d3efbb212569dea636b45c3eda572a06"}
Nov 03 00:17:40.605722 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.605663    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29" event={"ID":"da283988-fca2-4bde-8256-4589d9536e4b","Type":"ContainerDied","Data":"516ed4d2840070ebed59f2aae66fea0cfcfcab8a7bd8cd50045fd6b5e10edc0e"}
Nov 03 00:17:40.605722 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.605680    4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="516ed4d2840070ebed59f2aae66fea0cfcfcab8a7bd8cd50045fd6b5e10edc0e"
Nov 03 00:17:40.605722 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.605694    4294 scope.go:122] "RemoveContainer" containerID="d1735b7f2e08e253b049e20bb5e5d05d34d35673bc7c0a8d396e6fc6a9af6c8c"
Nov 03 00:17:40.621624 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.621600    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 03 00:17:40.670748 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.670710    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-istio-data\") pod \"da283988-fca2-4bde-8256-4589d9536e4b\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") "
Nov 03 00:17:40.670748 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.670757    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da283988-fca2-4bde-8256-4589d9536e4b-istio-token\" (UniqueName: \"kubernetes.io/projected/da283988-fca2-4bde-8256-4589d9536e4b-istio-token\") pod \"da283988-fca2-4bde-8256-4589d9536e4b\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") "
Nov 03 00:17:40.671055 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.670783    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da283988-fca2-4bde-8256-4589d9536e4b-kube-api-access-vjvjl\" (UniqueName: \"kubernetes.io/projected/da283988-fca2-4bde-8256-4589d9536e4b-kube-api-access-vjvjl\") pod \"da283988-fca2-4bde-8256-4589d9536e4b\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") "
Nov 03 00:17:40.671055 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.670818    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-workload-socket\") pod \"da283988-fca2-4bde-8256-4589d9536e4b\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") "
Nov 03 00:17:40.671055 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.670852    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da283988-fca2-4bde-8256-4589d9536e4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da283988-fca2-4bde-8256-4589d9536e4b-istio-podinfo\") pod \"da283988-fca2-4bde-8256-4589d9536e4b\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") "
Nov 03 00:17:40.671055 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.670912    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-istio-envoy\") pod \"da283988-fca2-4bde-8256-4589d9536e4b\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") "
Nov 03 00:17:40.671055 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.670942    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da283988-fca2-4bde-8256-4589d9536e4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da283988-fca2-4bde-8256-4589d9536e4b-istiod-ca-cert\") pod \"da283988-fca2-4bde-8256-4589d9536e4b\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") "
Nov 03 00:17:40.671055 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.671004    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-workload-certs\") pod \"da283988-fca2-4bde-8256-4589d9536e4b\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") "
Nov 03 00:17:40.671055 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.671034    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-istio-data" pod "da283988-fca2-4bde-8256-4589d9536e4b" (UID: "da283988-fca2-4bde-8256-4589d9536e4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:17:40.671452 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.671100    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-credential-socket\") pod \"da283988-fca2-4bde-8256-4589d9536e4b\" (UID: \"da283988-fca2-4bde-8256-4589d9536e4b\") "
Nov 03 00:17:40.671452 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.671233    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-workload-socket" pod "da283988-fca2-4bde-8256-4589d9536e4b" (UID: "da283988-fca2-4bde-8256-4589d9536e4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:17:40.671563 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.671440    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-workload-certs" pod "da283988-fca2-4bde-8256-4589d9536e4b" (UID: "da283988-fca2-4bde-8256-4589d9536e4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:17:40.671563 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.671475    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-credential-socket" pod "da283988-fca2-4bde-8256-4589d9536e4b" (UID: "da283988-fca2-4bde-8256-4589d9536e4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:17:40.671680 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.671592    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:40.671680 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.671615    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:40.671680 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.671632    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:40.671680 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.671646    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:40.671680 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.671653    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da283988-fca2-4bde-8256-4589d9536e4b-istiod-ca-cert" pod "da283988-fca2-4bde-8256-4589d9536e4b" (UID: "da283988-fca2-4bde-8256-4589d9536e4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:17:40.673241 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.673213    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da283988-fca2-4bde-8256-4589d9536e4b-kube-api-access-vjvjl" pod "da283988-fca2-4bde-8256-4589d9536e4b" (UID: "da283988-fca2-4bde-8256-4589d9536e4b"). InnerVolumeSpecName "kube-api-access-vjvjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:17:40.673366 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.673238    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da283988-fca2-4bde-8256-4589d9536e4b-istio-token" pod "da283988-fca2-4bde-8256-4589d9536e4b" (UID: "da283988-fca2-4bde-8256-4589d9536e4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:17:40.673366 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.673267    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-istio-envoy" pod "da283988-fca2-4bde-8256-4589d9536e4b" (UID: "da283988-fca2-4bde-8256-4589d9536e4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:17:40.673366 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.673291    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da283988-fca2-4bde-8256-4589d9536e4b-istio-podinfo" pod "da283988-fca2-4bde-8256-4589d9536e4b" (UID: "da283988-fca2-4bde-8256-4589d9536e4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:17:40.773122 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.773085    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da283988-fca2-4bde-8256-4589d9536e4b-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:40.773122 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.773120    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjvjl\" (UniqueName: \"kubernetes.io/projected/da283988-fca2-4bde-8256-4589d9536e4b-kube-api-access-vjvjl\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:40.773359 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.773134    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da283988-fca2-4bde-8256-4589d9536e4b-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:40.773359 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.773146    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da283988-fca2-4bde-8256-4589d9536e4b-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:40.773359 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:40.773160    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da283988-fca2-4bde-8256-4589d9536e4b-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:41.295194 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:41.295154    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj" podUID="edca56c1-37b2-4055-bb5a-d43f3d6cb47a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:17:41.613236 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:41.613157    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"
Nov 03 00:17:41.749286 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:17:41.749251    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:17:41.892231 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:41.892150    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"]
Nov 03 00:17:41.897387 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:41.897359    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3d8404-openshift-default-5fddcf4d77-bnx29"]
Nov 03 00:17:42.102477 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:42.102444    4294 scope.go:122] "RemoveContainer" containerID="54cb2277ba4e27b198d663aba7731387d3efbb212569dea636b45c3eda572a06"
Nov 03 00:17:42.546834 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:42.546801    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da283988-fca2-4bde-8256-4589d9536e4b" path="/var/lib/kubelet/pods/da283988-fca2-4bde-8256-4589d9536e4b/volumes"
Nov 03 00:17:43.748427 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:43.748384    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:48.427563 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:48.427508    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:49.094195 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:49.094162    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:49.335663 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:49.335630    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:50.447231 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:50.447197    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:51.295295 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:51.295254    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj" podUID="edca56c1-37b2-4055-bb5a-d43f3d6cb47a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:17:52.504688 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:52.504654    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:52.582006 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:52.581919    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj"
Nov 03 00:17:52.671255 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:52.671220    4294 generic.go:356] "Generic (PLEG): container finished" podID="edca56c1-37b2-4055-bb5a-d43f3d6cb47a" containerID="6f1b86e19226c1434e57926cf7a2538930933c6fae8775196e862ba76705a352" exitCode=0
Nov 03 00:17:52.671443 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:52.671281    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj"
Nov 03 00:17:52.671443 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:52.671308    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj" event={"ID":"edca56c1-37b2-4055-bb5a-d43f3d6cb47a","Type":"ContainerDied","Data":"6f1b86e19226c1434e57926cf7a2538930933c6fae8775196e862ba76705a352"}
Nov 03 00:17:52.671443 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:52.671414    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj" event={"ID":"edca56c1-37b2-4055-bb5a-d43f3d6cb47a","Type":"ContainerDied","Data":"b2e971caf676b7d11ba3f011ba00a78932b87f10d9476f95f0b2959eb13d2d47"}
Nov 03 00:17:52.671443 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:52.671440    4294 scope.go:122] "RemoveContainer" containerID="6f1b86e19226c1434e57926cf7a2538930933c6fae8775196e862ba76705a352"
Nov 03 00:17:52.680530 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:52.680511    4294 scope.go:122] "RemoveContainer" containerID="6f1b86e19226c1434e57926cf7a2538930933c6fae8775196e862ba76705a352"
Nov 03 00:17:52.680927 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:17:52.680907    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f1b86e19226c1434e57926cf7a2538930933c6fae8775196e862ba76705a352\": container with ID starting with 6f1b86e19226c1434e57926cf7a2538930933c6fae8775196e862ba76705a352 not found: ID does not exist" containerID="6f1b86e19226c1434e57926cf7a2538930933c6fae8775196e862ba76705a352"
Nov 03 00:17:52.680997 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:52.680937    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f1b86e19226c1434e57926cf7a2538930933c6fae8775196e862ba76705a352"} err="failed to get container status \"6f1b86e19226c1434e57926cf7a2538930933c6fae8775196e862ba76705a352\": rpc error: code = NotFound desc = could not find container \"6f1b86e19226c1434e57926cf7a2538930933c6fae8775196e862ba76705a352\": container with ID starting with 6f1b86e19226c1434e57926cf7a2538930933c6fae8775196e862ba76705a352 not found: ID does not exist"
Nov 03 00:17:52.707243 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:52.707214    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edca56c1-37b2-4055-bb5a-d43f3d6cb47a-kube-api-access-knd76\" (UniqueName: \"kubernetes.io/projected/edca56c1-37b2-4055-bb5a-d43f3d6cb47a-kube-api-access-knd76\") pod \"edca56c1-37b2-4055-bb5a-d43f3d6cb47a\" (UID: \"edca56c1-37b2-4055-bb5a-d43f3d6cb47a\") "
Nov 03 00:17:52.707435 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:52.707357    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edca56c1-37b2-4055-bb5a-d43f3d6cb47a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edca56c1-37b2-4055-bb5a-d43f3d6cb47a-plugins-config-volume\") pod \"edca56c1-37b2-4055-bb5a-d43f3d6cb47a\" (UID: \"edca56c1-37b2-4055-bb5a-d43f3d6cb47a\") "
Nov 03 00:17:52.707745 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:52.707721    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edca56c1-37b2-4055-bb5a-d43f3d6cb47a-plugins-config-volume" pod "edca56c1-37b2-4055-bb5a-d43f3d6cb47a" (UID: "edca56c1-37b2-4055-bb5a-d43f3d6cb47a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:17:52.709872 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:52.709844    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edca56c1-37b2-4055-bb5a-d43f3d6cb47a-kube-api-access-knd76" pod "edca56c1-37b2-4055-bb5a-d43f3d6cb47a" (UID: "edca56c1-37b2-4055-bb5a-d43f3d6cb47a"). InnerVolumeSpecName "kube-api-access-knd76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:17:52.809021 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:52.808960    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knd76\" (UniqueName: \"kubernetes.io/projected/edca56c1-37b2-4055-bb5a-d43f3d6cb47a-kube-api-access-knd76\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:52.809021 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:52.809016    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edca56c1-37b2-4055-bb5a-d43f3d6cb47a-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:17:53.091049 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:53.091005    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj"]
Nov 03 00:17:53.097412 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:53.097382    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7ca60f-84b5956795-nvmnj"]
Nov 03 00:17:53.185753 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:53.185718    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:54.546927 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:54.546883    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edca56c1-37b2-4055-bb5a-d43f3d6cb47a" path="/var/lib/kubelet/pods/edca56c1-37b2-4055-bb5a-d43f3d6cb47a/volumes"
Nov 03 00:17:55.156759 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:55.156720    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:56.955249 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:56.955211    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:57.310234 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:57.310196    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:58.490667 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:58.490631    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:58.709731 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:58.709695    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:59.023665 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:17:59.023628    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:02.286416 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:02.286384    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:02.499889 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:02.499850    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:03.839540 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:03.839508    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:04.147433 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:04.147331    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:04.394224 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:04.394191    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:04.762687 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:04.762656    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:09.024904 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:09.024868    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:09.709745 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:09.709708    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:09.726505 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:09.726466    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"]
Nov 03 00:18:09.929046 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:09.929012    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:11.750490 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:18:11.750446    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:18:13.850174 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:13.850140    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:16.004881 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:16.004846    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:18.509266 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:18.509233    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:22.890013 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:22.889957    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:23.196034 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:23.195948    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"]
Nov 03 00:18:23.300216 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:23.300181    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:23.743207 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:23.743174    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:24.746381 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:24.746317    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" podUID="845035ca-c55e-4e32-a19a-dc3617e3d638" containerName="main" containerID="cri-o://281569a0a2b76c0af4dfc908efb83b1f70c09f0f5ba313c3d0b82af2f638222f" gracePeriod=45
Nov 03 00:18:24.750683 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:24.750655    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" podUID="845035ca-c55e-4e32-a19a-dc3617e3d638" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": EOF"
Nov 03 00:18:24.998807 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:24.998753    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 03 00:18:25.149065 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.149036    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-tmp-dir\") pod \"845035ca-c55e-4e32-a19a-dc3617e3d638\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") "
Nov 03 00:18:25.149231 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.149089    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845035ca-c55e-4e32-a19a-dc3617e3d638-kube-api-access-mt7rp\" (UniqueName: \"kubernetes.io/projected/845035ca-c55e-4e32-a19a-dc3617e3d638-kube-api-access-mt7rp\") pod \"845035ca-c55e-4e32-a19a-dc3617e3d638\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") "
Nov 03 00:18:25.149231 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.149113    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-model-cache\") pod \"845035ca-c55e-4e32-a19a-dc3617e3d638\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") "
Nov 03 00:18:25.149231 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.149191    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-dshm\" (UniqueName: \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-dshm\") pod \"845035ca-c55e-4e32-a19a-dc3617e3d638\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") "
Nov 03 00:18:25.149389 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.149246    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/845035ca-c55e-4e32-a19a-dc3617e3d638-tls-certs\" (UniqueName: \"kubernetes.io/secret/845035ca-c55e-4e32-a19a-dc3617e3d638-tls-certs\") pod \"845035ca-c55e-4e32-a19a-dc3617e3d638\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") "
Nov 03 00:18:25.149389 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.149291    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-home\" (UniqueName: \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-home\") pod \"845035ca-c55e-4e32-a19a-dc3617e3d638\" (UID: \"845035ca-c55e-4e32-a19a-dc3617e3d638\") "
Nov 03 00:18:25.149389 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.149341    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-tmp-dir" pod "845035ca-c55e-4e32-a19a-dc3617e3d638" (UID: "845035ca-c55e-4e32-a19a-dc3617e3d638"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:18:25.149389 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.149377    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-model-cache" pod "845035ca-c55e-4e32-a19a-dc3617e3d638" (UID: "845035ca-c55e-4e32-a19a-dc3617e3d638"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:18:25.149625 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.149600    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-home" pod "845035ca-c55e-4e32-a19a-dc3617e3d638" (UID: "845035ca-c55e-4e32-a19a-dc3617e3d638"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:18:25.149760 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.149735    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:25.149897 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.149761    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:25.149897 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.149777    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:25.151401 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.151369    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-dshm" pod "845035ca-c55e-4e32-a19a-dc3617e3d638" (UID: "845035ca-c55e-4e32-a19a-dc3617e3d638"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:18:25.151500 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.151435    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845035ca-c55e-4e32-a19a-dc3617e3d638-kube-api-access-mt7rp" pod "845035ca-c55e-4e32-a19a-dc3617e3d638" (UID: "845035ca-c55e-4e32-a19a-dc3617e3d638"). InnerVolumeSpecName "kube-api-access-mt7rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:18:25.151500 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.151461    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/845035ca-c55e-4e32-a19a-dc3617e3d638-tls-certs" pod "845035ca-c55e-4e32-a19a-dc3617e3d638" (UID: "845035ca-c55e-4e32-a19a-dc3617e3d638"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:18:25.250516 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.250445    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/845035ca-c55e-4e32-a19a-dc3617e3d638-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:25.250516 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.250474    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt7rp\" (UniqueName: \"kubernetes.io/projected/845035ca-c55e-4e32-a19a-dc3617e3d638-kube-api-access-mt7rp\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:25.250516 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.250487    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/845035ca-c55e-4e32-a19a-dc3617e3d638-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:25.263571 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.263539    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k"]
Nov 03 00:18:25.364444 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.364413    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j"]
Nov 03 00:18:25.653938 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.653846    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:25.852595 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.852561    4294 generic.go:356] "Generic (PLEG): container finished" podID="845035ca-c55e-4e32-a19a-dc3617e3d638" containerID="281569a0a2b76c0af4dfc908efb83b1f70c09f0f5ba313c3d0b82af2f638222f" exitCode=0
Nov 03 00:18:25.852595 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.852599    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"
Nov 03 00:18:25.853100 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.852640    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" event={"ID":"845035ca-c55e-4e32-a19a-dc3617e3d638","Type":"ContainerDied","Data":"281569a0a2b76c0af4dfc908efb83b1f70c09f0f5ba313c3d0b82af2f638222f"}
Nov 03 00:18:25.853100 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.852679    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg" event={"ID":"845035ca-c55e-4e32-a19a-dc3617e3d638","Type":"ContainerDied","Data":"79bf2253e56b7ab77716fca065c511f430145da6b6577f50255af1720f0d4da7"}
Nov 03 00:18:25.853100 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.852697    4294 scope.go:122] "RemoveContainer" containerID="281569a0a2b76c0af4dfc908efb83b1f70c09f0f5ba313c3d0b82af2f638222f"
Nov 03 00:18:25.854566 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.854526    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:25.862467 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.862449    4294 scope.go:122] "RemoveContainer" containerID="281569a0a2b76c0af4dfc908efb83b1f70c09f0f5ba313c3d0b82af2f638222f"
Nov 03 00:18:25.862723 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:18:25.862706    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281569a0a2b76c0af4dfc908efb83b1f70c09f0f5ba313c3d0b82af2f638222f\": container with ID starting with 281569a0a2b76c0af4dfc908efb83b1f70c09f0f5ba313c3d0b82af2f638222f not found: ID does not exist" containerID="281569a0a2b76c0af4dfc908efb83b1f70c09f0f5ba313c3d0b82af2f638222f"
Nov 03 00:18:25.862772 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:25.862729    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281569a0a2b76c0af4dfc908efb83b1f70c09f0f5ba313c3d0b82af2f638222f"} err="failed to get container status \"281569a0a2b76c0af4dfc908efb83b1f70c09f0f5ba313c3d0b82af2f638222f\": rpc error: code = NotFound desc = could not find container \"281569a0a2b76c0af4dfc908efb83b1f70c09f0f5ba313c3d0b82af2f638222f\": container with ID starting with 281569a0a2b76c0af4dfc908efb83b1f70c09f0f5ba313c3d0b82af2f638222f not found: ID does not exist"
Nov 03 00:18:26.016893 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:26.016859    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"]
Nov 03 00:18:26.019778 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:26.019758    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-14b321/simulator-w3-14b321-kserve-5758674467-z72pg"]
Nov 03 00:18:26.549864 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:26.549831    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="845035ca-c55e-4e32-a19a-dc3617e3d638" path="/var/lib/kubelet/pods/845035ca-c55e-4e32-a19a-dc3617e3d638/volumes"
Nov 03 00:18:26.577234 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:26.577206    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"]
Nov 03 00:18:26.670880 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:26.670845    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:27.108546 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:27.108494    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:28.294719 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:28.294680    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:29.691957 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:29.691923    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:30.264093 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:30.264026    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k" podUID="3c5057a9-8986-4bc3-8a01-2866a8aa09ca" containerName="payload-processing" containerID="cri-o://28e1a01703ed8fe960870caafa4e48832446909a6173b40e0f3122a2b13f448a" gracePeriod=25
Nov 03 00:18:30.365330 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:30.365274    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j" podUID="705e82db-2105-4a14-8990-5f0b1322da1d" containerName="payload-pre-processing" containerID="cri-o://88545ebb05732b51cfd2ef00abc9b36f0413bef82c91db081641dc320fa0adf4" gracePeriod=25
Nov 03 00:18:32.233644 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:32.233609    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:33.425332 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:33.425271    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:34.757294 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:34.757248    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j" podUID="705e82db-2105-4a14-8990-5f0b1322da1d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:18:34.757710 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:34.757326    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k" podUID="3c5057a9-8986-4bc3-8a01-2866a8aa09ca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:18:36.817439 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:36.817404    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:38.216725 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.216686    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7" podUID="989580e5-edec-4c8b-97b8-c9e105dc3306" containerName="main" containerID="cri-o://7e00fcd6b3b6c9cf08739e07536012d1c79b0c1f771b91b7d63381005dee2323" gracePeriod=45
Nov 03 00:18:38.458131 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.458107    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:18:38.493253 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.493226    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-home\" (UniqueName: \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-home\") pod \"989580e5-edec-4c8b-97b8-c9e105dc3306\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") "
Nov 03 00:18:38.493449 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.493271    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-dshm\" (UniqueName: \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-dshm\") pod \"989580e5-edec-4c8b-97b8-c9e105dc3306\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") "
Nov 03 00:18:38.493449 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.493302    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/989580e5-edec-4c8b-97b8-c9e105dc3306-tls-certs\" (UniqueName: \"kubernetes.io/secret/989580e5-edec-4c8b-97b8-c9e105dc3306-tls-certs\") pod \"989580e5-edec-4c8b-97b8-c9e105dc3306\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") "
Nov 03 00:18:38.493449 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.493417    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-tmp-dir\") pod \"989580e5-edec-4c8b-97b8-c9e105dc3306\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") "
Nov 03 00:18:38.493648 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.493515    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-model-cache\") pod \"989580e5-edec-4c8b-97b8-c9e105dc3306\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") "
Nov 03 00:18:38.493648 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.493526    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-home" pod "989580e5-edec-4c8b-97b8-c9e105dc3306" (UID: "989580e5-edec-4c8b-97b8-c9e105dc3306"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:18:38.493648 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.493563    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989580e5-edec-4c8b-97b8-c9e105dc3306-kube-api-access-bqzvt\" (UniqueName: \"kubernetes.io/projected/989580e5-edec-4c8b-97b8-c9e105dc3306-kube-api-access-bqzvt\") pod \"989580e5-edec-4c8b-97b8-c9e105dc3306\" (UID: \"989580e5-edec-4c8b-97b8-c9e105dc3306\") "
Nov 03 00:18:38.493787 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.493740    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-tmp-dir" pod "989580e5-edec-4c8b-97b8-c9e105dc3306" (UID: "989580e5-edec-4c8b-97b8-c9e105dc3306"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:18:38.493855 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.493822    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-model-cache" pod "989580e5-edec-4c8b-97b8-c9e105dc3306" (UID: "989580e5-edec-4c8b-97b8-c9e105dc3306"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:18:38.494137 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.494044    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:38.494571 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.494140    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:38.494571 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.494156    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:38.495832 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.495800    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989580e5-edec-4c8b-97b8-c9e105dc3306-kube-api-access-bqzvt" pod "989580e5-edec-4c8b-97b8-c9e105dc3306" (UID: "989580e5-edec-4c8b-97b8-c9e105dc3306"). InnerVolumeSpecName "kube-api-access-bqzvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:18:38.495832 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.495817    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/989580e5-edec-4c8b-97b8-c9e105dc3306-tls-certs" pod "989580e5-edec-4c8b-97b8-c9e105dc3306" (UID: "989580e5-edec-4c8b-97b8-c9e105dc3306"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:18:38.496059 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.495867    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-dshm" pod "989580e5-edec-4c8b-97b8-c9e105dc3306" (UID: "989580e5-edec-4c8b-97b8-c9e105dc3306"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:18:38.509344 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.509318    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:38.594686 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.594656    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqzvt\" (UniqueName: \"kubernetes.io/projected/989580e5-edec-4c8b-97b8-c9e105dc3306-kube-api-access-bqzvt\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:38.594686 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.594686    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/989580e5-edec-4c8b-97b8-c9e105dc3306-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:38.594876 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.594699    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/989580e5-edec-4c8b-97b8-c9e105dc3306-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:38.928768 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.928679    4294 generic.go:356] "Generic (PLEG): container finished" podID="989580e5-edec-4c8b-97b8-c9e105dc3306" containerID="7e00fcd6b3b6c9cf08739e07536012d1c79b0c1f771b91b7d63381005dee2323" exitCode=0
Nov 03 00:18:38.928768 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.928726    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"
Nov 03 00:18:38.929053 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.928772    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7" event={"ID":"989580e5-edec-4c8b-97b8-c9e105dc3306","Type":"ContainerDied","Data":"7e00fcd6b3b6c9cf08739e07536012d1c79b0c1f771b91b7d63381005dee2323"}
Nov 03 00:18:38.929053 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.928821    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7" event={"ID":"989580e5-edec-4c8b-97b8-c9e105dc3306","Type":"ContainerDied","Data":"d7191343daf6a03fe2f7f1c65cb738b7916e4c76df28b17a4cd9fc88da56691e"}
Nov 03 00:18:38.929053 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.928844    4294 scope.go:122] "RemoveContainer" containerID="7e00fcd6b3b6c9cf08739e07536012d1c79b0c1f771b91b7d63381005dee2323"
Nov 03 00:18:38.937552 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.937533    4294 scope.go:122] "RemoveContainer" containerID="7e00fcd6b3b6c9cf08739e07536012d1c79b0c1f771b91b7d63381005dee2323"
Nov 03 00:18:38.937827 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:18:38.937808    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e00fcd6b3b6c9cf08739e07536012d1c79b0c1f771b91b7d63381005dee2323\": container with ID starting with 7e00fcd6b3b6c9cf08739e07536012d1c79b0c1f771b91b7d63381005dee2323 not found: ID does not exist" containerID="7e00fcd6b3b6c9cf08739e07536012d1c79b0c1f771b91b7d63381005dee2323"
Nov 03 00:18:38.937895 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:38.937842    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e00fcd6b3b6c9cf08739e07536012d1c79b0c1f771b91b7d63381005dee2323"} err="failed to get container status \"7e00fcd6b3b6c9cf08739e07536012d1c79b0c1f771b91b7d63381005dee2323\": rpc error: code = NotFound desc = could not find container \"7e00fcd6b3b6c9cf08739e07536012d1c79b0c1f771b91b7d63381005dee2323\": container with ID starting with 7e00fcd6b3b6c9cf08739e07536012d1c79b0c1f771b91b7d63381005dee2323 not found: ID does not exist"
Nov 03 00:18:39.101667 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:39.101630    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"]
Nov 03 00:18:39.139197 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:39.139170    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7"]
Nov 03 00:18:39.396506 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:39.396456    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-b4d53e/sub-test-model-b4d53e-kserve-685dcdff5f-whtc7" podUID="989580e5-edec-4c8b-97b8-c9e105dc3306" containerName="main" probeResult="failure" output="Get \"https://10.133.0.203:8000/ready\": context deadline exceeded"
Nov 03 00:18:40.546923 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:40.546894    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="989580e5-edec-4c8b-97b8-c9e105dc3306" path="/var/lib/kubelet/pods/989580e5-edec-4c8b-97b8-c9e105dc3306/volumes"
Nov 03 00:18:41.597156 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.597116    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn" podUID="a976b1a0-de58-4e35-a698-aadd48bd47cc" containerName="main" containerID="cri-o://82cf9c3d854ce4022bf241c6ced896315b34456c7b13dff4930bd684c5c81ccb" gracePeriod=45
Nov 03 00:18:41.750967 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:18:41.750927    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:18:41.850926 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.850862    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:18:41.921032 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.920998    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a976b1a0-de58-4e35-a698-aadd48bd47cc-tls-certs\" (UniqueName: \"kubernetes.io/secret/a976b1a0-de58-4e35-a698-aadd48bd47cc-tls-certs\") pod \"a976b1a0-de58-4e35-a698-aadd48bd47cc\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") "
Nov 03 00:18:41.921032 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.921037    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-home\" (UniqueName: \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-home\") pod \"a976b1a0-de58-4e35-a698-aadd48bd47cc\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") "
Nov 03 00:18:41.921308 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.921070    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a976b1a0-de58-4e35-a698-aadd48bd47cc-kube-api-access-lht8m\" (UniqueName: \"kubernetes.io/projected/a976b1a0-de58-4e35-a698-aadd48bd47cc-kube-api-access-lht8m\") pod \"a976b1a0-de58-4e35-a698-aadd48bd47cc\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") "
Nov 03 00:18:41.921308 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.921153    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-model-cache\") pod \"a976b1a0-de58-4e35-a698-aadd48bd47cc\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") "
Nov 03 00:18:41.921308 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.921192    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-tmp-dir\") pod \"a976b1a0-de58-4e35-a698-aadd48bd47cc\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") "
Nov 03 00:18:41.921467 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.921348    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-home" pod "a976b1a0-de58-4e35-a698-aadd48bd47cc" (UID: "a976b1a0-de58-4e35-a698-aadd48bd47cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:18:41.921467 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.921400    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-model-cache" pod "a976b1a0-de58-4e35-a698-aadd48bd47cc" (UID: "a976b1a0-de58-4e35-a698-aadd48bd47cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:18:41.921553 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.921471    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-tmp-dir" pod "a976b1a0-de58-4e35-a698-aadd48bd47cc" (UID: "a976b1a0-de58-4e35-a698-aadd48bd47cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:18:41.921553 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.921498    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:41.921553 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.921510    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:41.923225 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.923207    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a976b1a0-de58-4e35-a698-aadd48bd47cc-tls-certs" pod "a976b1a0-de58-4e35-a698-aadd48bd47cc" (UID: "a976b1a0-de58-4e35-a698-aadd48bd47cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:18:41.923309 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.923289    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a976b1a0-de58-4e35-a698-aadd48bd47cc-kube-api-access-lht8m" pod "a976b1a0-de58-4e35-a698-aadd48bd47cc" (UID: "a976b1a0-de58-4e35-a698-aadd48bd47cc"). InnerVolumeSpecName "kube-api-access-lht8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:18:41.944347 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.944319    4294 generic.go:356] "Generic (PLEG): container finished" podID="a976b1a0-de58-4e35-a698-aadd48bd47cc" containerID="82cf9c3d854ce4022bf241c6ced896315b34456c7b13dff4930bd684c5c81ccb" exitCode=0
Nov 03 00:18:41.944448 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.944363    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"
Nov 03 00:18:41.944448 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.944399    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn" event={"ID":"a976b1a0-de58-4e35-a698-aadd48bd47cc","Type":"ContainerDied","Data":"82cf9c3d854ce4022bf241c6ced896315b34456c7b13dff4930bd684c5c81ccb"}
Nov 03 00:18:41.944448 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.944439    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn" event={"ID":"a976b1a0-de58-4e35-a698-aadd48bd47cc","Type":"ContainerDied","Data":"b98b825d317c7747c68be03895214343c93d193aa951e6ce012f17803b519da4"}
Nov 03 00:18:41.944572 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.944456    4294 scope.go:122] "RemoveContainer" containerID="82cf9c3d854ce4022bf241c6ced896315b34456c7b13dff4930bd684c5c81ccb"
Nov 03 00:18:41.953008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.952965    4294 scope.go:122] "RemoveContainer" containerID="82cf9c3d854ce4022bf241c6ced896315b34456c7b13dff4930bd684c5c81ccb"
Nov 03 00:18:41.953249 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:18:41.953232    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82cf9c3d854ce4022bf241c6ced896315b34456c7b13dff4930bd684c5c81ccb\": container with ID starting with 82cf9c3d854ce4022bf241c6ced896315b34456c7b13dff4930bd684c5c81ccb not found: ID does not exist" containerID="82cf9c3d854ce4022bf241c6ced896315b34456c7b13dff4930bd684c5c81ccb"
Nov 03 00:18:41.953297 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:41.953258    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82cf9c3d854ce4022bf241c6ced896315b34456c7b13dff4930bd684c5c81ccb"} err="failed to get container status \"82cf9c3d854ce4022bf241c6ced896315b34456c7b13dff4930bd684c5c81ccb\": rpc error: code = NotFound desc = could not find container \"82cf9c3d854ce4022bf241c6ced896315b34456c7b13dff4930bd684c5c81ccb\": container with ID starting with 82cf9c3d854ce4022bf241c6ced896315b34456c7b13dff4930bd684c5c81ccb not found: ID does not exist"
Nov 03 00:18:42.022410 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:42.022377    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-dshm\") pod \"a976b1a0-de58-4e35-a698-aadd48bd47cc\" (UID: \"a976b1a0-de58-4e35-a698-aadd48bd47cc\") "
Nov 03 00:18:42.022659 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:42.022647    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:42.022723 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:42.022661    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a976b1a0-de58-4e35-a698-aadd48bd47cc-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:42.022723 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:42.022671    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lht8m\" (UniqueName: \"kubernetes.io/projected/a976b1a0-de58-4e35-a698-aadd48bd47cc-kube-api-access-lht8m\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:42.024445 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:42.024425    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-dshm" pod "a976b1a0-de58-4e35-a698-aadd48bd47cc" (UID: "a976b1a0-de58-4e35-a698-aadd48bd47cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:18:42.124247 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:42.124161    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a976b1a0-de58-4e35-a698-aadd48bd47cc-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:42.358337 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:42.358305    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"]
Nov 03 00:18:42.406246 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:42.406181    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a51378/sub-test-model-a51378-kserve-58d599dbc-t9wdn"]
Nov 03 00:18:42.546961 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:42.546931    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a976b1a0-de58-4e35-a698-aadd48bd47cc" path="/var/lib/kubelet/pods/a976b1a0-de58-4e35-a698-aadd48bd47cc/volumes"
Nov 03 00:18:42.578814 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:42.578785    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:44.307090 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:44.307044    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:44.756503 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:44.756466    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k" podUID="3c5057a9-8986-4bc3-8a01-2866a8aa09ca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:18:44.756653 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:44.756516    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j" podUID="705e82db-2105-4a14-8990-5f0b1322da1d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:18:47.146143 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.146110    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:47.363780 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.363739    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:47.936349 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936320    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"]
Nov 03 00:18:47.936772 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936759    4294 cpu_manager.go:414] "containerMap: removing container" podUID="785b24ed-8d15-410f-b7d4-02549e6e2201" containerName="main"
Nov 03 00:18:47.936819 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936773    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="785b24ed-8d15-410f-b7d4-02549e6e2201" containerName="main"
Nov 03 00:18:47.936819 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936791    4294 cpu_manager.go:414] "containerMap: removing container" podUID="a2210e3e-ee40-4277-8865-93acdf8e81a4" containerName="maas-api"
Nov 03 00:18:47.936819 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936799    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2210e3e-ee40-4277-8865-93acdf8e81a4" containerName="maas-api"
Nov 03 00:18:47.936819 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936812    4294 cpu_manager.go:414] "containerMap: removing container" podUID="989580e5-edec-4c8b-97b8-c9e105dc3306" containerName="main"
Nov 03 00:18:47.936819 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936817    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="989580e5-edec-4c8b-97b8-c9e105dc3306" containerName="main"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936826    4294 cpu_manager.go:414] "containerMap: removing container" podUID="da283988-fca2-4bde-8256-4589d9536e4b" containerName="istio-proxy"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936830    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da283988-fca2-4bde-8256-4589d9536e4b" containerName="istio-proxy"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936838    4294 cpu_manager.go:414] "containerMap: removing container" podUID="959479c1-5aad-4927-9d79-384ef72ebf9e" containerName="payload-processing"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936843    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="959479c1-5aad-4927-9d79-384ef72ebf9e" containerName="payload-processing"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936852    4294 cpu_manager.go:414] "containerMap: removing container" podUID="924707d4-bd50-4ce6-9d59-b9abf9d41f64" containerName="payload-pre-processing"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936857    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924707d4-bd50-4ce6-9d59-b9abf9d41f64" containerName="payload-pre-processing"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936863    4294 cpu_manager.go:414] "containerMap: removing container" podUID="78e187a5-e905-410b-a2c5-ddced7693786" containerName="payload-processing"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936868    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e187a5-e905-410b-a2c5-ddced7693786" containerName="payload-processing"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936885    4294 cpu_manager.go:414] "containerMap: removing container" podUID="36cb1d31-8467-441f-90bf-07ba20b0f3c6" containerName="payload-processing"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936890    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36cb1d31-8467-441f-90bf-07ba20b0f3c6" containerName="payload-processing"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936901    4294 cpu_manager.go:414] "containerMap: removing container" podUID="845035ca-c55e-4e32-a19a-dc3617e3d638" containerName="main"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936905    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845035ca-c55e-4e32-a19a-dc3617e3d638" containerName="main"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936913    4294 cpu_manager.go:414] "containerMap: removing container" podUID="b716a68d-1d7e-48d6-98a5-6fdc5472cac9" containerName="payload-pre-processing"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936918    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b716a68d-1d7e-48d6-98a5-6fdc5472cac9" containerName="payload-pre-processing"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936925    4294 cpu_manager.go:414] "containerMap: removing container" podUID="95ad5c99-119f-46ca-8664-8f2878dbca94" containerName="payload-pre-processing"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936929    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ad5c99-119f-46ca-8664-8f2878dbca94" containerName="payload-pre-processing"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936937    4294 cpu_manager.go:414] "containerMap: removing container" podUID="a976b1a0-de58-4e35-a698-aadd48bd47cc" containerName="main"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936946    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a976b1a0-de58-4e35-a698-aadd48bd47cc" containerName="main"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936957    4294 cpu_manager.go:414] "containerMap: removing container" podUID="edca56c1-37b2-4055-bb5a-d43f3d6cb47a" containerName="payload-pre-processing"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936964    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edca56c1-37b2-4055-bb5a-d43f3d6cb47a" containerName="payload-pre-processing"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936989    4294 cpu_manager.go:414] "containerMap: removing container" podUID="b86a51a3-351e-46c3-ad35-68eb39e6ee8d" containerName="test-curl-7427-a3ac02"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.936994    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86a51a3-351e-46c3-ad35-68eb39e6ee8d" containerName="test-curl-7427-a3ac02"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.937000    4294 cpu_manager.go:414] "containerMap: removing container" podUID="da283988-fca2-4bde-8256-4589d9536e4b" containerName="istio-proxy"
Nov 03 00:18:47.937008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.937004    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da283988-fca2-4bde-8256-4589d9536e4b" containerName="istio-proxy"
Nov 03 00:18:47.937811 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.937073    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="924707d4-bd50-4ce6-9d59-b9abf9d41f64" containerName="payload-pre-processing"
Nov 03 00:18:47.937811 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.937082    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e187a5-e905-410b-a2c5-ddced7693786" containerName="payload-processing"
Nov 03 00:18:47.937811 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.937089    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="36cb1d31-8467-441f-90bf-07ba20b0f3c6" containerName="payload-processing"
Nov 03 00:18:47.937811 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.937095    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="845035ca-c55e-4e32-a19a-dc3617e3d638" containerName="main"
Nov 03 00:18:47.937811 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.937101    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="b716a68d-1d7e-48d6-98a5-6fdc5472cac9" containerName="payload-pre-processing"
Nov 03 00:18:47.937811 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.937108    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="95ad5c99-119f-46ca-8664-8f2878dbca94" containerName="payload-pre-processing"
Nov 03 00:18:47.937811 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.937117    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a976b1a0-de58-4e35-a698-aadd48bd47cc" containerName="main"
Nov 03 00:18:47.937811 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.937125    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="edca56c1-37b2-4055-bb5a-d43f3d6cb47a" containerName="payload-pre-processing"
Nov 03 00:18:47.937811 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.937131    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="b86a51a3-351e-46c3-ad35-68eb39e6ee8d" containerName="test-curl-7427-a3ac02"
Nov 03 00:18:47.937811 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.937136    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="da283988-fca2-4bde-8256-4589d9536e4b" containerName="istio-proxy"
Nov 03 00:18:47.937811 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.937141    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="785b24ed-8d15-410f-b7d4-02549e6e2201" containerName="main"
Nov 03 00:18:47.937811 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.937147    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2210e3e-ee40-4277-8865-93acdf8e81a4" containerName="maas-api"
Nov 03 00:18:47.937811 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.937152    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="989580e5-edec-4c8b-97b8-c9e105dc3306" containerName="main"
Nov 03 00:18:47.937811 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.937156    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="da283988-fca2-4bde-8256-4589d9536e4b" containerName="istio-proxy"
Nov 03 00:18:47.937811 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.937162    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="959479c1-5aad-4927-9d79-384ef72ebf9e" containerName="payload-processing"
Nov 03 00:18:47.942112 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.942095    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:47.980276 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.980248    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-data\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:47.980427 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.980318    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-token\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:47.980427 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.980382    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-envoy\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:47.980504 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.980441    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-credential-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:47.980504 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.980487    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-podinfo\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:47.980572 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.980550    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istiod-ca-cert\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:47.980608 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.980581    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-workload-certs\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:47.980642 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.980607    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crgrg\" (UniqueName: \"kubernetes.io/projected/8868fdae-c6a7-4eb7-bef0-255f6586e34c-kube-api-access-crgrg\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:47.980727 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.980707    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-workload-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:47.996205 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:47.996185    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-14b321-openshift-default-dockercfg-ssm6b\""
Nov 03 00:18:48.042539 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.042505    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"]
Nov 03 00:18:48.081840 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.081814    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-data\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:48.082015 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.081869    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-token\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:48.082015 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.081893    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-envoy\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:48.082015 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.081911    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-credential-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:48.082015 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.081938    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-podinfo\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:48.082235 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.082094    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istiod-ca-cert\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:48.082235 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.082154    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-workload-certs\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:48.082235 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.082192    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crgrg\" (UniqueName: \"kubernetes.io/projected/8868fdae-c6a7-4eb7-bef0-255f6586e34c-kube-api-access-crgrg\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:48.082399 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.082243    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-workload-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:48.082399 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.082251    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-data\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:48.082399 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.082307    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-credential-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:48.082579 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.082534    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-workload-certs\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:48.082579 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.082552    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-workload-socket\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:48.082797 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.082778    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istiod-ca-cert\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:48.084112 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.084094    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-envoy\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:48.084450 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.084432    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-podinfo\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:48.141856 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.141824    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crgrg\" (UniqueName: \"kubernetes.io/projected/8868fdae-c6a7-4eb7-bef0-255f6586e34c-kube-api-access-crgrg\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:48.142727 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.142707    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-token\") pod \"e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") " pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:48.239391 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.239303    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:48.255205 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.255180    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:48.719760 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:18:48.719727    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8868fdae_c6a7_4eb7_bef0_255f6586e34c.slice/crio-c5a88b4bceada7d7d76034ee7439c173d5faefbb1cfa8411e02e05a3ea52af07 WatchSource:0}: Error finding container c5a88b4bceada7d7d76034ee7439c173d5faefbb1cfa8411e02e05a3ea52af07: Status 404 returned error can't find the container with id c5a88b4bceada7d7d76034ee7439c173d5faefbb1cfa8411e02e05a3ea52af07
Nov 03 00:18:48.721603 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.721581    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"]
Nov 03 00:18:48.722185 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.722153    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:48.722299 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.722229    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:48.722299 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.722273    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:48.981096 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.981008    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99" event={"ID":"8868fdae-c6a7-4eb7-bef0-255f6586e34c","Type":"ContainerStarted","Data":"24b6238f6249bbd4ebd4b27860ed0286b1ee37546c917e5efeb645649d823cf8"}
Nov 03 00:18:48.981096 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:48.981060    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99" event={"ID":"8868fdae-c6a7-4eb7-bef0-255f6586e34c","Type":"ContainerStarted","Data":"c5a88b4bceada7d7d76034ee7439c173d5faefbb1cfa8411e02e05a3ea52af07"}
Nov 03 00:18:49.105771 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:49.105704    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99" podStartSLOduration=2.105684666 podStartE2EDuration="2.105684666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:49.104768204 +0000 UTC m=+5168.673122997" watchObservedRunningTime="2026-11-03 00:18:49.105684666 +0000 UTC m=+5168.674039454"
Nov 03 00:18:49.134821 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:49.134787    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:49.271004 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:49.270916    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:49.271548 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:49.271066    4294 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 03 00:18:49.271548 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:49.271110    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99" podUID="8868fdae-c6a7-4eb7-bef0-255f6586e34c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 03 00:18:50.271695 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.270343    4294 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 03 00:18:50.271695 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.270407    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99" podUID="8868fdae-c6a7-4eb7-bef0-255f6586e34c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 03 00:18:50.546046 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.546020    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k"
Nov 03 00:18:50.607479 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.607453    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c5057a9-8986-4bc3-8a01-2866a8aa09ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c5057a9-8986-4bc3-8a01-2866a8aa09ca-plugins-config-volume\") pod \"3c5057a9-8986-4bc3-8a01-2866a8aa09ca\" (UID: \"3c5057a9-8986-4bc3-8a01-2866a8aa09ca\") "
Nov 03 00:18:50.607639 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.607523    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c5057a9-8986-4bc3-8a01-2866a8aa09ca-kube-api-access-wf4vc\" (UniqueName: \"kubernetes.io/projected/3c5057a9-8986-4bc3-8a01-2866a8aa09ca-kube-api-access-wf4vc\") pod \"3c5057a9-8986-4bc3-8a01-2866a8aa09ca\" (UID: \"3c5057a9-8986-4bc3-8a01-2866a8aa09ca\") "
Nov 03 00:18:50.607905 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.607879    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c5057a9-8986-4bc3-8a01-2866a8aa09ca-plugins-config-volume" pod "3c5057a9-8986-4bc3-8a01-2866a8aa09ca" (UID: "3c5057a9-8986-4bc3-8a01-2866a8aa09ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:18:50.609889 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.609861    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5057a9-8986-4bc3-8a01-2866a8aa09ca-kube-api-access-wf4vc" pod "3c5057a9-8986-4bc3-8a01-2866a8aa09ca" (UID: "3c5057a9-8986-4bc3-8a01-2866a8aa09ca"). InnerVolumeSpecName "kube-api-access-wf4vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:18:50.625170 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.625152    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j"
Nov 03 00:18:50.708614 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.708585    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705e82db-2105-4a14-8990-5f0b1322da1d-kube-api-access-7hq79\" (UniqueName: \"kubernetes.io/projected/705e82db-2105-4a14-8990-5f0b1322da1d-kube-api-access-7hq79\") pod \"705e82db-2105-4a14-8990-5f0b1322da1d\" (UID: \"705e82db-2105-4a14-8990-5f0b1322da1d\") "
Nov 03 00:18:50.708794 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.708674    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/705e82db-2105-4a14-8990-5f0b1322da1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705e82db-2105-4a14-8990-5f0b1322da1d-plugins-config-volume\") pod \"705e82db-2105-4a14-8990-5f0b1322da1d\" (UID: \"705e82db-2105-4a14-8990-5f0b1322da1d\") "
Nov 03 00:18:50.708917 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.708903    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c5057a9-8986-4bc3-8a01-2866a8aa09ca-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:50.709004 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.708917    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf4vc\" (UniqueName: \"kubernetes.io/projected/3c5057a9-8986-4bc3-8a01-2866a8aa09ca-kube-api-access-wf4vc\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:50.709058 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.709034    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705e82db-2105-4a14-8990-5f0b1322da1d-plugins-config-volume" pod "705e82db-2105-4a14-8990-5f0b1322da1d" (UID: "705e82db-2105-4a14-8990-5f0b1322da1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:18:50.710672 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.710653    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705e82db-2105-4a14-8990-5f0b1322da1d-kube-api-access-7hq79" pod "705e82db-2105-4a14-8990-5f0b1322da1d" (UID: "705e82db-2105-4a14-8990-5f0b1322da1d"). InnerVolumeSpecName "kube-api-access-7hq79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:18:50.779053 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.778817    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:50.809464 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.809434    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/705e82db-2105-4a14-8990-5f0b1322da1d-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:50.809464 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.809463    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hq79\" (UniqueName: \"kubernetes.io/projected/705e82db-2105-4a14-8990-5f0b1322da1d-kube-api-access-7hq79\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:18:50.992335 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.992291    4294 generic.go:356] "Generic (PLEG): container finished" podID="3c5057a9-8986-4bc3-8a01-2866a8aa09ca" containerID="28e1a01703ed8fe960870caafa4e48832446909a6173b40e0f3122a2b13f448a" exitCode=0
Nov 03 00:18:50.992702 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.992354    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k"
Nov 03 00:18:50.992702 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.992368    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k" event={"ID":"3c5057a9-8986-4bc3-8a01-2866a8aa09ca","Type":"ContainerDied","Data":"28e1a01703ed8fe960870caafa4e48832446909a6173b40e0f3122a2b13f448a"}
Nov 03 00:18:50.992702 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.992442    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k" event={"ID":"3c5057a9-8986-4bc3-8a01-2866a8aa09ca","Type":"ContainerDied","Data":"163bf0a5391dbb9dda9239ab8ce407a14ff14861636a557147720b1953421832"}
Nov 03 00:18:50.992702 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.992464    4294 scope.go:122] "RemoveContainer" containerID="28e1a01703ed8fe960870caafa4e48832446909a6173b40e0f3122a2b13f448a"
Nov 03 00:18:50.994059 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.994031    4294 generic.go:356] "Generic (PLEG): container finished" podID="705e82db-2105-4a14-8990-5f0b1322da1d" containerID="88545ebb05732b51cfd2ef00abc9b36f0413bef82c91db081641dc320fa0adf4" exitCode=0
Nov 03 00:18:50.994201 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.994094    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j"
Nov 03 00:18:50.994201 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.994105    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j" event={"ID":"705e82db-2105-4a14-8990-5f0b1322da1d","Type":"ContainerDied","Data":"88545ebb05732b51cfd2ef00abc9b36f0413bef82c91db081641dc320fa0adf4"}
Nov 03 00:18:50.994201 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:50.994146    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j" event={"ID":"705e82db-2105-4a14-8990-5f0b1322da1d","Type":"ContainerDied","Data":"ceb27adb42ed1d14528d38dca1b9256054a441604bb588a03281649efb19b3dc"}
Nov 03 00:18:51.002785 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:51.002667    4294 scope.go:122] "RemoveContainer" containerID="28e1a01703ed8fe960870caafa4e48832446909a6173b40e0f3122a2b13f448a"
Nov 03 00:18:51.003038 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:18:51.003012    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e1a01703ed8fe960870caafa4e48832446909a6173b40e0f3122a2b13f448a\": container with ID starting with 28e1a01703ed8fe960870caafa4e48832446909a6173b40e0f3122a2b13f448a not found: ID does not exist" containerID="28e1a01703ed8fe960870caafa4e48832446909a6173b40e0f3122a2b13f448a"
Nov 03 00:18:51.003138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:51.003046    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e1a01703ed8fe960870caafa4e48832446909a6173b40e0f3122a2b13f448a"} err="failed to get container status \"28e1a01703ed8fe960870caafa4e48832446909a6173b40e0f3122a2b13f448a\": rpc error: code = NotFound desc = could not find container \"28e1a01703ed8fe960870caafa4e48832446909a6173b40e0f3122a2b13f448a\": container with ID starting with 28e1a01703ed8fe960870caafa4e48832446909a6173b40e0f3122a2b13f448a not found: ID does not exist"
Nov 03 00:18:51.003138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:51.003063    4294 scope.go:122] "RemoveContainer" containerID="88545ebb05732b51cfd2ef00abc9b36f0413bef82c91db081641dc320fa0adf4"
Nov 03 00:18:51.011278 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:51.011260    4294 scope.go:122] "RemoveContainer" containerID="88545ebb05732b51cfd2ef00abc9b36f0413bef82c91db081641dc320fa0adf4"
Nov 03 00:18:51.011563 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:18:51.011541    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88545ebb05732b51cfd2ef00abc9b36f0413bef82c91db081641dc320fa0adf4\": container with ID starting with 88545ebb05732b51cfd2ef00abc9b36f0413bef82c91db081641dc320fa0adf4 not found: ID does not exist" containerID="88545ebb05732b51cfd2ef00abc9b36f0413bef82c91db081641dc320fa0adf4"
Nov 03 00:18:51.011631 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:51.011569    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88545ebb05732b51cfd2ef00abc9b36f0413bef82c91db081641dc320fa0adf4"} err="failed to get container status \"88545ebb05732b51cfd2ef00abc9b36f0413bef82c91db081641dc320fa0adf4\": rpc error: code = NotFound desc = could not find container \"88545ebb05732b51cfd2ef00abc9b36f0413bef82c91db081641dc320fa0adf4\": container with ID starting with 88545ebb05732b51cfd2ef00abc9b36f0413bef82c91db081641dc320fa0adf4 not found: ID does not exist"
Nov 03 00:18:51.152607 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:51.152527    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j"]
Nov 03 00:18:51.164005 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:51.160835    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-14b321-588679f5c9-8l89j"]
Nov 03 00:18:51.261004 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:51.257345    4294 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 03 00:18:51.261004 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:51.257402    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99" podUID="8868fdae-c6a7-4eb7-bef0-255f6586e34c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 03 00:18:51.287023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:51.286974    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k"]
Nov 03 00:18:51.289301 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:51.289279    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-14b321-595f4b44c9-cv87k"]
Nov 03 00:18:52.271003 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:52.261410    4294 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 03 00:18:52.271003 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:52.261471    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99" podUID="8868fdae-c6a7-4eb7-bef0-255f6586e34c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 03 00:18:52.546716 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:52.546642    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c5057a9-8986-4bc3-8a01-2866a8aa09ca" path="/var/lib/kubelet/pods/3c5057a9-8986-4bc3-8a01-2866a8aa09ca/volumes"
Nov 03 00:18:52.547121 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:52.547080    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="705e82db-2105-4a14-8990-5f0b1322da1d" path="/var/lib/kubelet/pods/705e82db-2105-4a14-8990-5f0b1322da1d/volumes"
Nov 03 00:18:53.273008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:53.269339    4294 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 03 00:18:53.273008 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:53.269400    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99" podUID="8868fdae-c6a7-4eb7-bef0-255f6586e34c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 03 00:18:53.968628 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:53.968545    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:54.263004 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:54.259375    4294 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 03 00:18:54.263004 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:54.259441    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99" podUID="8868fdae-c6a7-4eb7-bef0-255f6586e34c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 03 00:18:54.711316 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:54.711211    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:54.986273 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:54.986161    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:55.255657 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:55.255629    4294 patch_prober.go:29] interesting pod/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Nov 03 00:18:55.255832 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:55.255685    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99" podUID="8868fdae-c6a7-4eb7-bef0-255f6586e34c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Nov 03 00:18:56.261086 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:56.261053    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:56.261749 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:56.261714    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:56.262582 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:56.262566    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:18:56.304910 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:56.304884    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:56.755654 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:56.755624    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"]
Nov 03 00:18:56.844690 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:56.844634    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:57.026412 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:57.026340    4294 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99" secret="" err="secret \"e2e-worker-w3-14b321-openshift-default-dockercfg-ssm6b\" not found"
Nov 03 00:18:57.076831 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:57.076804    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:57.309728 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:57.309614    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:57.533315 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:57.533275    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:58.031892 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:58.031851    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99" podUID="8868fdae-c6a7-4eb7-bef0-255f6586e34c" containerName="istio-proxy" containerID="cri-o://24b6238f6249bbd4ebd4b27860ed0286b1ee37546c917e5efeb645649d823cf8" gracePeriod=30
Nov 03 00:18:59.213080 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:18:59.213047    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:03.280099 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.280075    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:19:03.440783 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.440713    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-envoy\") pod \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") "
Nov 03 00:19:03.440783 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.440754    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istiod-ca-cert\") pod \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") "
Nov 03 00:19:03.440783 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.440781    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-workload-socket\") pod \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") "
Nov 03 00:19:03.441093 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.440820    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-data\") pod \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") "
Nov 03 00:19:03.441093 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.440840    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8868fdae-c6a7-4eb7-bef0-255f6586e34c-kube-api-access-crgrg\" (UniqueName: \"kubernetes.io/projected/8868fdae-c6a7-4eb7-bef0-255f6586e34c-kube-api-access-crgrg\") pod \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") "
Nov 03 00:19:03.441093 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.440866    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-podinfo\") pod \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") "
Nov 03 00:19:03.441093 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.440907    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-token\" (UniqueName: \"kubernetes.io/projected/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-token\") pod \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") "
Nov 03 00:19:03.441093 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.440941    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-credential-socket\") pod \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") "
Nov 03 00:19:03.441093 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.441044    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-workload-socket" pod "8868fdae-c6a7-4eb7-bef0-255f6586e34c" (UID: "8868fdae-c6a7-4eb7-bef0-255f6586e34c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:19:03.441093 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.441054    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-workload-certs\") pod \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\" (UID: \"8868fdae-c6a7-4eb7-bef0-255f6586e34c\") "
Nov 03 00:19:03.441540 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.441181    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-data" pod "8868fdae-c6a7-4eb7-bef0-255f6586e34c" (UID: "8868fdae-c6a7-4eb7-bef0-255f6586e34c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:19:03.441540 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.441186    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istiod-ca-cert" pod "8868fdae-c6a7-4eb7-bef0-255f6586e34c" (UID: "8868fdae-c6a7-4eb7-bef0-255f6586e34c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:19:03.441540 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.441254    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-workload-certs" pod "8868fdae-c6a7-4eb7-bef0-255f6586e34c" (UID: "8868fdae-c6a7-4eb7-bef0-255f6586e34c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:19:03.441540 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.441421    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-credential-socket" pod "8868fdae-c6a7-4eb7-bef0-255f6586e34c" (UID: "8868fdae-c6a7-4eb7-bef0-255f6586e34c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:19:03.441540 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.441448    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:19:03.441540 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.441471    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:19:03.441540 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.441486    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:19:03.441540 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.441499    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:19:03.443066 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.443044    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-envoy" pod "8868fdae-c6a7-4eb7-bef0-255f6586e34c" (UID: "8868fdae-c6a7-4eb7-bef0-255f6586e34c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:19:03.443177 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.443071    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-podinfo" pod "8868fdae-c6a7-4eb7-bef0-255f6586e34c" (UID: "8868fdae-c6a7-4eb7-bef0-255f6586e34c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:19:03.443177 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.443078    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8868fdae-c6a7-4eb7-bef0-255f6586e34c-kube-api-access-crgrg" pod "8868fdae-c6a7-4eb7-bef0-255f6586e34c" (UID: "8868fdae-c6a7-4eb7-bef0-255f6586e34c"). InnerVolumeSpecName "kube-api-access-crgrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:19:03.443177 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.443128    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-token" pod "8868fdae-c6a7-4eb7-bef0-255f6586e34c" (UID: "8868fdae-c6a7-4eb7-bef0-255f6586e34c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:19:03.542974 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.542948    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:19:03.542974 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.542972    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crgrg\" (UniqueName: \"kubernetes.io/projected/8868fdae-c6a7-4eb7-bef0-255f6586e34c-kube-api-access-crgrg\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:19:03.543173 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.543001    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:19:03.543173 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.543010    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8868fdae-c6a7-4eb7-bef0-255f6586e34c-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:19:03.543173 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.543018    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8868fdae-c6a7-4eb7-bef0-255f6586e34c-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:19:03.897784 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:03.897752    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:04.060936 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:04.060906    4294 generic.go:356] "Generic (PLEG): container finished" podID="8868fdae-c6a7-4eb7-bef0-255f6586e34c" containerID="24b6238f6249bbd4ebd4b27860ed0286b1ee37546c917e5efeb645649d823cf8" exitCode=0
Nov 03 00:19:04.061128 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:04.060953    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99" event={"ID":"8868fdae-c6a7-4eb7-bef0-255f6586e34c","Type":"ContainerDied","Data":"24b6238f6249bbd4ebd4b27860ed0286b1ee37546c917e5efeb645649d823cf8"}
Nov 03 00:19:04.061128 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:04.060955    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"
Nov 03 00:19:04.061128 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:04.061011    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99" event={"ID":"8868fdae-c6a7-4eb7-bef0-255f6586e34c","Type":"ContainerDied","Data":"c5a88b4bceada7d7d76034ee7439c173d5faefbb1cfa8411e02e05a3ea52af07"}
Nov 03 00:19:04.061128 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:04.061033    4294 scope.go:122] "RemoveContainer" containerID="24b6238f6249bbd4ebd4b27860ed0286b1ee37546c917e5efeb645649d823cf8"
Nov 03 00:19:04.070657 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:04.070643    4294 scope.go:122] "RemoveContainer" containerID="24b6238f6249bbd4ebd4b27860ed0286b1ee37546c917e5efeb645649d823cf8"
Nov 03 00:19:04.070904 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:19:04.070887    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24b6238f6249bbd4ebd4b27860ed0286b1ee37546c917e5efeb645649d823cf8\": container with ID starting with 24b6238f6249bbd4ebd4b27860ed0286b1ee37546c917e5efeb645649d823cf8 not found: ID does not exist" containerID="24b6238f6249bbd4ebd4b27860ed0286b1ee37546c917e5efeb645649d823cf8"
Nov 03 00:19:04.070944 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:04.070911    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24b6238f6249bbd4ebd4b27860ed0286b1ee37546c917e5efeb645649d823cf8"} err="failed to get container status \"24b6238f6249bbd4ebd4b27860ed0286b1ee37546c917e5efeb645649d823cf8\": rpc error: code = NotFound desc = could not find container \"24b6238f6249bbd4ebd4b27860ed0286b1ee37546c917e5efeb645649d823cf8\": container with ID starting with 24b6238f6249bbd4ebd4b27860ed0286b1ee37546c917e5efeb645649d823cf8 not found: ID does not exist"
Nov 03 00:19:04.225960 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:04.225894    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"]
Nov 03 00:19:04.273020 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:04.272994    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-14b321-openshift-default-c4f7c678f-dvr99"]
Nov 03 00:19:04.546314 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:04.546288    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8868fdae-c6a7-4eb7-bef0-255f6586e34c" path="/var/lib/kubelet/pods/8868fdae-c6a7-4eb7-bef0-255f6586e34c/volumes"
Nov 03 00:19:08.785674 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:08.785644    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:10.035694 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:10.035650    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8"]
Nov 03 00:19:10.113004 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:10.112963    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt"]
Nov 03 00:19:10.443152 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:10.443068    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:10.656765 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:10.656726    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:11.057738 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:11.057702    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:11.748173 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:19:11.748141    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:19:13.378588 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:13.378558    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:15.091163 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:15.091082    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8" podUID="723b2ff2-4870-4de1-b820-6813a12699b7" containerName="payload-processing" containerID="cri-o://641736d12eaeee6ad944b7500388811a1bb711c7118cc95f122d7828641cbbeb" gracePeriod=25
Nov 03 00:19:15.113776 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:15.113741    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt" podUID="4079e7c5-252d-40d6-ac37-67dde32884f6" containerName="payload-pre-processing" containerID="cri-o://9b665ad8870f945e75a7c07c45d77ed5d2fb569cc5dc993a263084cfece37027" gracePeriod=25
Nov 03 00:19:15.191677 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:15.191643    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:17.739997 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:17.739958    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:17.956946 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:17.956910    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:19.366413 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:19.366379    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:19.871603 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:19.871555    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:20.203437 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:20.203359    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:20.421799 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:20.421762    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8" podUID="723b2ff2-4870-4de1-b820-6813a12699b7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:19:20.422221 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:20.421768    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt" podUID="4079e7c5-252d-40d6-ac37-67dde32884f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:19:20.606563 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:20.606535    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:25.097594 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:25.097563    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:26.871333 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:26.871299    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:27.847080 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:27.847049    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6"]
Nov 03 00:19:28.135617 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:28.135540    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:28.358218 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:28.358177    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:29.143233 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:29.143205    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"]
Nov 03 00:19:29.143684 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:29.143425    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt" podUID="00c80137-cc6a-48e5-9e02-184ee47f56da" containerName="maas-api" containerID="cri-o://30db606424c8fb22e8bbba46b97472afba6c26d51490da1f31bd0b10938f0850" gracePeriod=30
Nov 03 00:19:29.286971 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:29.286936    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:29.397483 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:29.397405    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"
Nov 03 00:19:29.485531 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:29.485491    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c80137-cc6a-48e5-9e02-184ee47f56da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c80137-cc6a-48e5-9e02-184ee47f56da-openshift-service-ca\") pod \"00c80137-cc6a-48e5-9e02-184ee47f56da\" (UID: \"00c80137-cc6a-48e5-9e02-184ee47f56da\") "
Nov 03 00:19:29.485716 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:29.485540    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c80137-cc6a-48e5-9e02-184ee47f56da-kube-api-access-hwr79\" (UniqueName: \"kubernetes.io/projected/00c80137-cc6a-48e5-9e02-184ee47f56da-kube-api-access-hwr79\") pod \"00c80137-cc6a-48e5-9e02-184ee47f56da\" (UID: \"00c80137-cc6a-48e5-9e02-184ee47f56da\") "
Nov 03 00:19:29.485716 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:29.485656    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00c80137-cc6a-48e5-9e02-184ee47f56da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c80137-cc6a-48e5-9e02-184ee47f56da-maas-api-tls\") pod \"00c80137-cc6a-48e5-9e02-184ee47f56da\" (UID: \"00c80137-cc6a-48e5-9e02-184ee47f56da\") "
Nov 03 00:19:29.485807 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:29.485734    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00c80137-cc6a-48e5-9e02-184ee47f56da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c80137-cc6a-48e5-9e02-184ee47f56da-metrics-certs\") pod \"00c80137-cc6a-48e5-9e02-184ee47f56da\" (UID: \"00c80137-cc6a-48e5-9e02-184ee47f56da\") "
Nov 03 00:19:29.485874 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:29.485849    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c80137-cc6a-48e5-9e02-184ee47f56da-openshift-service-ca" pod "00c80137-cc6a-48e5-9e02-184ee47f56da" (UID: "00c80137-cc6a-48e5-9e02-184ee47f56da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:19:29.486058 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:29.486034    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00c80137-cc6a-48e5-9e02-184ee47f56da-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:19:29.487774 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:29.487742    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c80137-cc6a-48e5-9e02-184ee47f56da-maas-api-tls" pod "00c80137-cc6a-48e5-9e02-184ee47f56da" (UID: "00c80137-cc6a-48e5-9e02-184ee47f56da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:19:29.487889 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:29.487797    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00c80137-cc6a-48e5-9e02-184ee47f56da-metrics-certs" pod "00c80137-cc6a-48e5-9e02-184ee47f56da" (UID: "00c80137-cc6a-48e5-9e02-184ee47f56da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:19:29.487889 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:29.487840    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c80137-cc6a-48e5-9e02-184ee47f56da-kube-api-access-hwr79" pod "00c80137-cc6a-48e5-9e02-184ee47f56da" (UID: "00c80137-cc6a-48e5-9e02-184ee47f56da"). InnerVolumeSpecName "kube-api-access-hwr79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:19:29.586702 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:29.586665    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwr79\" (UniqueName: \"kubernetes.io/projected/00c80137-cc6a-48e5-9e02-184ee47f56da-kube-api-access-hwr79\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:19:29.586702 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:29.586695    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00c80137-cc6a-48e5-9e02-184ee47f56da-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:19:29.586702 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:29.586705    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00c80137-cc6a-48e5-9e02-184ee47f56da-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:19:30.187135 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:30.187104    4294 generic.go:356] "Generic (PLEG): container finished" podID="00c80137-cc6a-48e5-9e02-184ee47f56da" containerID="30db606424c8fb22e8bbba46b97472afba6c26d51490da1f31bd0b10938f0850" exitCode=0
Nov 03 00:19:30.187664 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:30.187148    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"
Nov 03 00:19:30.187664 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:30.187191    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt" event={"ID":"00c80137-cc6a-48e5-9e02-184ee47f56da","Type":"ContainerDied","Data":"30db606424c8fb22e8bbba46b97472afba6c26d51490da1f31bd0b10938f0850"}
Nov 03 00:19:30.187664 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:30.187254    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt" event={"ID":"00c80137-cc6a-48e5-9e02-184ee47f56da","Type":"ContainerDied","Data":"4ddcf8bc46df3a4e9858219483d162ed8c297be5804c823f084f4349eddae038"}
Nov 03 00:19:30.187664 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:30.187292    4294 scope.go:122] "RemoveContainer" containerID="30db606424c8fb22e8bbba46b97472afba6c26d51490da1f31bd0b10938f0850"
Nov 03 00:19:30.196633 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:30.196618    4294 scope.go:122] "RemoveContainer" containerID="30db606424c8fb22e8bbba46b97472afba6c26d51490da1f31bd0b10938f0850"
Nov 03 00:19:30.196871 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:19:30.196855    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30db606424c8fb22e8bbba46b97472afba6c26d51490da1f31bd0b10938f0850\": container with ID starting with 30db606424c8fb22e8bbba46b97472afba6c26d51490da1f31bd0b10938f0850 not found: ID does not exist" containerID="30db606424c8fb22e8bbba46b97472afba6c26d51490da1f31bd0b10938f0850"
Nov 03 00:19:30.196923 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:30.196879    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30db606424c8fb22e8bbba46b97472afba6c26d51490da1f31bd0b10938f0850"} err="failed to get container status \"30db606424c8fb22e8bbba46b97472afba6c26d51490da1f31bd0b10938f0850\": rpc error: code = NotFound desc = could not find container \"30db606424c8fb22e8bbba46b97472afba6c26d51490da1f31bd0b10938f0850\": container with ID starting with 30db606424c8fb22e8bbba46b97472afba6c26d51490da1f31bd0b10938f0850 not found: ID does not exist"
Nov 03 00:19:30.348302 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:30.348268    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"]
Nov 03 00:19:30.353785 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:30.353764    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a51378-866f897c59-rzgbt"]
Nov 03 00:19:30.422269 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:30.422238    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt" podUID="4079e7c5-252d-40d6-ac37-67dde32884f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:19:30.422404 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:30.422245    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8" podUID="723b2ff2-4870-4de1-b820-6813a12699b7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:19:30.546464 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:30.546434    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c80137-cc6a-48e5-9e02-184ee47f56da" path="/var/lib/kubelet/pods/00c80137-cc6a-48e5-9e02-184ee47f56da/volumes"
Nov 03 00:19:31.213623 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:31.213586    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:32.597251 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:32.597212    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:32.847830 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:32.847731    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6" podUID="fba0cc63-6a7a-474c-b6e4-c411f912eb46" containerName="payload-pre-processing" containerID="cri-o://d66244dc81c334afe93fc8d4ea6e574e4fda691f1ed8665a263b1cdb42b6e6be" gracePeriod=25
Nov 03 00:19:35.214033 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:35.214004    4294 generic.go:356] "Generic (PLEG): container finished" podID="723b2ff2-4870-4de1-b820-6813a12699b7" containerID="641736d12eaeee6ad944b7500388811a1bb711c7118cc95f122d7828641cbbeb" exitCode=0
Nov 03 00:19:35.214392 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:35.214065    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8" event={"ID":"723b2ff2-4870-4de1-b820-6813a12699b7","Type":"ContainerDied","Data":"641736d12eaeee6ad944b7500388811a1bb711c7118cc95f122d7828641cbbeb"}
Nov 03 00:19:35.257635 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:35.257599    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6" podUID="fba0cc63-6a7a-474c-b6e4-c411f912eb46" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:19:35.375192 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:35.375167    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8"
Nov 03 00:19:35.378058 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:35.378039    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt"
Nov 03 00:19:35.442659 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:35.442637    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/723b2ff2-4870-4de1-b820-6813a12699b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/723b2ff2-4870-4de1-b820-6813a12699b7-plugins-config-volume\") pod \"723b2ff2-4870-4de1-b820-6813a12699b7\" (UID: \"723b2ff2-4870-4de1-b820-6813a12699b7\") "
Nov 03 00:19:35.442796 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:35.442680    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/723b2ff2-4870-4de1-b820-6813a12699b7-kube-api-access-swzx7\" (UniqueName: \"kubernetes.io/projected/723b2ff2-4870-4de1-b820-6813a12699b7-kube-api-access-swzx7\") pod \"723b2ff2-4870-4de1-b820-6813a12699b7\" (UID: \"723b2ff2-4870-4de1-b820-6813a12699b7\") "
Nov 03 00:19:35.442796 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:35.442717    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4079e7c5-252d-40d6-ac37-67dde32884f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4079e7c5-252d-40d6-ac37-67dde32884f6-plugins-config-volume\") pod \"4079e7c5-252d-40d6-ac37-67dde32884f6\" (UID: \"4079e7c5-252d-40d6-ac37-67dde32884f6\") "
Nov 03 00:19:35.442913 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:35.442794    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4079e7c5-252d-40d6-ac37-67dde32884f6-kube-api-access-zjm4x\" (UniqueName: \"kubernetes.io/projected/4079e7c5-252d-40d6-ac37-67dde32884f6-kube-api-access-zjm4x\") pod \"4079e7c5-252d-40d6-ac37-67dde32884f6\" (UID: \"4079e7c5-252d-40d6-ac37-67dde32884f6\") "
Nov 03 00:19:35.443101 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:35.443073    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4079e7c5-252d-40d6-ac37-67dde32884f6-plugins-config-volume" pod "4079e7c5-252d-40d6-ac37-67dde32884f6" (UID: "4079e7c5-252d-40d6-ac37-67dde32884f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:19:35.443101 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:35.443075    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/723b2ff2-4870-4de1-b820-6813a12699b7-plugins-config-volume" pod "723b2ff2-4870-4de1-b820-6813a12699b7" (UID: "723b2ff2-4870-4de1-b820-6813a12699b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:19:35.444846 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:35.444824    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723b2ff2-4870-4de1-b820-6813a12699b7-kube-api-access-swzx7" pod "723b2ff2-4870-4de1-b820-6813a12699b7" (UID: "723b2ff2-4870-4de1-b820-6813a12699b7"). InnerVolumeSpecName "kube-api-access-swzx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:19:35.444936 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:35.444832    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4079e7c5-252d-40d6-ac37-67dde32884f6-kube-api-access-zjm4x" pod "4079e7c5-252d-40d6-ac37-67dde32884f6" (UID: "4079e7c5-252d-40d6-ac37-67dde32884f6"). InnerVolumeSpecName "kube-api-access-zjm4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:19:35.543811 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:35.543780    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/723b2ff2-4870-4de1-b820-6813a12699b7-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:19:35.543811 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:35.543806    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swzx7\" (UniqueName: \"kubernetes.io/projected/723b2ff2-4870-4de1-b820-6813a12699b7-kube-api-access-swzx7\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:19:35.544037 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:35.543823    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4079e7c5-252d-40d6-ac37-67dde32884f6-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:19:35.544037 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:35.543832    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjm4x\" (UniqueName: \"kubernetes.io/projected/4079e7c5-252d-40d6-ac37-67dde32884f6-kube-api-access-zjm4x\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:19:35.872377 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:35.872328    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:36.219580 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:36.219501    4294 generic.go:356] "Generic (PLEG): container finished" podID="4079e7c5-252d-40d6-ac37-67dde32884f6" containerID="9b665ad8870f945e75a7c07c45d77ed5d2fb569cc5dc993a263084cfece37027" exitCode=0
Nov 03 00:19:36.219580 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:36.219548    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt"
Nov 03 00:19:36.220154 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:36.219591    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt" event={"ID":"4079e7c5-252d-40d6-ac37-67dde32884f6","Type":"ContainerDied","Data":"9b665ad8870f945e75a7c07c45d77ed5d2fb569cc5dc993a263084cfece37027"}
Nov 03 00:19:36.220154 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:36.219648    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt" event={"ID":"4079e7c5-252d-40d6-ac37-67dde32884f6","Type":"ContainerDied","Data":"1710a70fbb6f606313128ea41831026074160f4cf91b92a145bdaba849a36399"}
Nov 03 00:19:36.220154 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:36.219679    4294 scope.go:122] "RemoveContainer" containerID="9b665ad8870f945e75a7c07c45d77ed5d2fb569cc5dc993a263084cfece37027"
Nov 03 00:19:36.221089 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:36.221070    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8" event={"ID":"723b2ff2-4870-4de1-b820-6813a12699b7","Type":"ContainerDied","Data":"6336b0f88a7e1258af52879472b9d2c6a19ea88d43c2f8ee029735bc81c0011d"}
Nov 03 00:19:36.221166 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:36.221124    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8"
Nov 03 00:19:36.230206 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:36.230185    4294 scope.go:122] "RemoveContainer" containerID="9b665ad8870f945e75a7c07c45d77ed5d2fb569cc5dc993a263084cfece37027"
Nov 03 00:19:36.230487 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:19:36.230470    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b665ad8870f945e75a7c07c45d77ed5d2fb569cc5dc993a263084cfece37027\": container with ID starting with 9b665ad8870f945e75a7c07c45d77ed5d2fb569cc5dc993a263084cfece37027 not found: ID does not exist" containerID="9b665ad8870f945e75a7c07c45d77ed5d2fb569cc5dc993a263084cfece37027"
Nov 03 00:19:36.230577 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:36.230497    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b665ad8870f945e75a7c07c45d77ed5d2fb569cc5dc993a263084cfece37027"} err="failed to get container status \"9b665ad8870f945e75a7c07c45d77ed5d2fb569cc5dc993a263084cfece37027\": rpc error: code = NotFound desc = could not find container \"9b665ad8870f945e75a7c07c45d77ed5d2fb569cc5dc993a263084cfece37027\": container with ID starting with 9b665ad8870f945e75a7c07c45d77ed5d2fb569cc5dc993a263084cfece37027 not found: ID does not exist"
Nov 03 00:19:36.230577 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:36.230522    4294 scope.go:122] "RemoveContainer" containerID="641736d12eaeee6ad944b7500388811a1bb711c7118cc95f122d7828641cbbeb"
Nov 03 00:19:36.378950 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:36.378917    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8"]
Nov 03 00:19:36.426521 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:36.426494    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b4d53e-76b6b85b7-srth8"]
Nov 03 00:19:36.491597 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:36.491573    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt"]
Nov 03 00:19:36.494392 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:36.494371    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b4d53e-b4d8dc675-fsmlt"]
Nov 03 00:19:36.546266 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:36.546237    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4079e7c5-252d-40d6-ac37-67dde32884f6" path="/var/lib/kubelet/pods/4079e7c5-252d-40d6-ac37-67dde32884f6/volumes"
Nov 03 00:19:36.546619 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:36.546607    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="723b2ff2-4870-4de1-b820-6813a12699b7" path="/var/lib/kubelet/pods/723b2ff2-4870-4de1-b820-6813a12699b7/volumes"
Nov 03 00:19:41.747414 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:19:41.747355    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:19:42.017820 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:42.017741    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:45.257628 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:45.257590    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6" podUID="fba0cc63-6a7a-474c-b6e4-c411f912eb46" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:19:46.799722 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:46.799687    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:48.137964 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:48.137915    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:48.332031 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:48.332000    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:49.624130 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:49.624100    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:50.165881 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:50.165848    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:50.613782 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:50.613746    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:51.106884 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:51.106856    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:51.760933 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:51.760902    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:53.094411 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:53.094389    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6"
Nov 03 00:19:53.207423 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:53.207278    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fba0cc63-6a7a-474c-b6e4-c411f912eb46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba0cc63-6a7a-474c-b6e4-c411f912eb46-plugins-config-volume\") pod \"fba0cc63-6a7a-474c-b6e4-c411f912eb46\" (UID: \"fba0cc63-6a7a-474c-b6e4-c411f912eb46\") "
Nov 03 00:19:53.207423 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:53.207384    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba0cc63-6a7a-474c-b6e4-c411f912eb46-kube-api-access-4ldwh\" (UniqueName: \"kubernetes.io/projected/fba0cc63-6a7a-474c-b6e4-c411f912eb46-kube-api-access-4ldwh\") pod \"fba0cc63-6a7a-474c-b6e4-c411f912eb46\" (UID: \"fba0cc63-6a7a-474c-b6e4-c411f912eb46\") "
Nov 03 00:19:53.207749 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:53.207674    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba0cc63-6a7a-474c-b6e4-c411f912eb46-plugins-config-volume" pod "fba0cc63-6a7a-474c-b6e4-c411f912eb46" (UID: "fba0cc63-6a7a-474c-b6e4-c411f912eb46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:19:53.209666 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:53.209639    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba0cc63-6a7a-474c-b6e4-c411f912eb46-kube-api-access-4ldwh" pod "fba0cc63-6a7a-474c-b6e4-c411f912eb46" (UID: "fba0cc63-6a7a-474c-b6e4-c411f912eb46"). InnerVolumeSpecName "kube-api-access-4ldwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:19:53.300583 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:53.300549    4294 generic.go:356] "Generic (PLEG): container finished" podID="fba0cc63-6a7a-474c-b6e4-c411f912eb46" containerID="d66244dc81c334afe93fc8d4ea6e574e4fda691f1ed8665a263b1cdb42b6e6be" exitCode=0
Nov 03 00:19:53.300734 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:53.300589    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6" event={"ID":"fba0cc63-6a7a-474c-b6e4-c411f912eb46","Type":"ContainerDied","Data":"d66244dc81c334afe93fc8d4ea6e574e4fda691f1ed8665a263b1cdb42b6e6be"}
Nov 03 00:19:53.300734 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:53.300630    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6" event={"ID":"fba0cc63-6a7a-474c-b6e4-c411f912eb46","Type":"ContainerDied","Data":"b7c69854072fd63d3bde528f76136c5aebe3942e91d7fd111a80ab9079c4dbe5"}
Nov 03 00:19:53.300734 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:53.300652    4294 scope.go:122] "RemoveContainer" containerID="d66244dc81c334afe93fc8d4ea6e574e4fda691f1ed8665a263b1cdb42b6e6be"
Nov 03 00:19:53.300734 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:53.300594    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6"
Nov 03 00:19:53.308339 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:53.308269    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ldwh\" (UniqueName: \"kubernetes.io/projected/fba0cc63-6a7a-474c-b6e4-c411f912eb46-kube-api-access-4ldwh\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:19:53.308339 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:53.308297    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fba0cc63-6a7a-474c-b6e4-c411f912eb46-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:19:53.309422 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:53.309405    4294 scope.go:122] "RemoveContainer" containerID="d66244dc81c334afe93fc8d4ea6e574e4fda691f1ed8665a263b1cdb42b6e6be"
Nov 03 00:19:53.309663 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:19:53.309642    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d66244dc81c334afe93fc8d4ea6e574e4fda691f1ed8665a263b1cdb42b6e6be\": container with ID starting with d66244dc81c334afe93fc8d4ea6e574e4fda691f1ed8665a263b1cdb42b6e6be not found: ID does not exist" containerID="d66244dc81c334afe93fc8d4ea6e574e4fda691f1ed8665a263b1cdb42b6e6be"
Nov 03 00:19:53.309708 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:53.309671    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d66244dc81c334afe93fc8d4ea6e574e4fda691f1ed8665a263b1cdb42b6e6be"} err="failed to get container status \"d66244dc81c334afe93fc8d4ea6e574e4fda691f1ed8665a263b1cdb42b6e6be\": rpc error: code = NotFound desc = could not find container \"d66244dc81c334afe93fc8d4ea6e574e4fda691f1ed8665a263b1cdb42b6e6be\": container with ID starting with d66244dc81c334afe93fc8d4ea6e574e4fda691f1ed8665a263b1cdb42b6e6be not found: ID does not exist"
Nov 03 00:19:53.459352 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:53.459262    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6"]
Nov 03 00:19:53.465098 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:53.465075    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a51378-77f9dd7999-kjsk6"]
Nov 03 00:19:54.545737 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:54.545707    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fba0cc63-6a7a-474c-b6e4-c411f912eb46" path="/var/lib/kubelet/pods/fba0cc63-6a7a-474c-b6e4-c411f912eb46/volumes"
Nov 03 00:19:55.548065 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:55.548031    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"]
Nov 03 00:19:55.709474 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:55.709441    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:56.667962 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:56.667932    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:57.444816 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:57.444784    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:57.630699 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:19:57.630670    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:01.270802 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:01.270770    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:03.881956 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:03.881914    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:03.996951 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:03.996912    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b"]
Nov 03 00:20:04.146263 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:04.146179    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr"]
Nov 03 00:20:04.415438 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:04.415336    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:04.644676 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:04.644641    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:08.997819 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:08.997752    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b" podUID="6f480410-faab-4712-b042-4e02d5759d11" containerName="payload-processing" containerID="cri-o://65abc4b4e8e5fb7a304380ca64c9681f8d95670734fe8e711b4bc062ae61a9fe" gracePeriod=25
Nov 03 00:20:09.146521 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:09.146477    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr" podUID="82d7bfa3-6d24-4821-bde2-5272ab59b0c3" containerName="payload-pre-processing" containerID="cri-o://e80be45d5d272679e8ac8c1712067ba42a7c3ab8a6ed4ce73ec212d018b37751" gracePeriod=25
Nov 03 00:20:09.612225 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:09.612185    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:10.567108 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:10.567040    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9" podUID="43e3d0d6-1a8e-4536-bc46-b5785124405a" containerName="main" containerID="cri-o://43911fcc2bc6f85a5cab5a2d21c83dc1b186416ec975aaa39a5a3e07c584cfab" gracePeriod=45
Nov 03 00:20:10.812809 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:10.812779    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 03 00:20:10.871542 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:10.871455    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e3d0d6-1a8e-4536-bc46-b5785124405a-kube-api-access-fphlq\" (UniqueName: \"kubernetes.io/projected/43e3d0d6-1a8e-4536-bc46-b5785124405a-kube-api-access-fphlq\") pod \"43e3d0d6-1a8e-4536-bc46-b5785124405a\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") "
Nov 03 00:20:10.871723 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:10.871600    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-dshm\") pod \"43e3d0d6-1a8e-4536-bc46-b5785124405a\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") "
Nov 03 00:20:10.871723 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:10.871669    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-tmp-dir\") pod \"43e3d0d6-1a8e-4536-bc46-b5785124405a\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") "
Nov 03 00:20:10.871723 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:10.871696    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43e3d0d6-1a8e-4536-bc46-b5785124405a-tls-certs\" (UniqueName: \"kubernetes.io/secret/43e3d0d6-1a8e-4536-bc46-b5785124405a-tls-certs\") pod \"43e3d0d6-1a8e-4536-bc46-b5785124405a\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") "
Nov 03 00:20:10.871723 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:10.871720    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-home\" (UniqueName: \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-home\") pod \"43e3d0d6-1a8e-4536-bc46-b5785124405a\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") "
Nov 03 00:20:10.871948 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:10.871753    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-model-cache\") pod \"43e3d0d6-1a8e-4536-bc46-b5785124405a\" (UID: \"43e3d0d6-1a8e-4536-bc46-b5785124405a\") "
Nov 03 00:20:10.872021 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:10.871943    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-tmp-dir" pod "43e3d0d6-1a8e-4536-bc46-b5785124405a" (UID: "43e3d0d6-1a8e-4536-bc46-b5785124405a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:20:10.872021 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:10.871953    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-home" pod "43e3d0d6-1a8e-4536-bc46-b5785124405a" (UID: "43e3d0d6-1a8e-4536-bc46-b5785124405a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:20:10.872125 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:10.872046    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-model-cache" pod "43e3d0d6-1a8e-4536-bc46-b5785124405a" (UID: "43e3d0d6-1a8e-4536-bc46-b5785124405a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:20:10.872181 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:10.872163    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:20:10.872233 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:10.872186    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:20:10.872233 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:10.872200    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:20:10.873826 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:10.873798    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-dshm" pod "43e3d0d6-1a8e-4536-bc46-b5785124405a" (UID: "43e3d0d6-1a8e-4536-bc46-b5785124405a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:20:10.873960 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:10.873912    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43e3d0d6-1a8e-4536-bc46-b5785124405a-tls-certs" pod "43e3d0d6-1a8e-4536-bc46-b5785124405a" (UID: "43e3d0d6-1a8e-4536-bc46-b5785124405a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:20:10.874042 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:10.874018    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e3d0d6-1a8e-4536-bc46-b5785124405a-kube-api-access-fphlq" pod "43e3d0d6-1a8e-4536-bc46-b5785124405a" (UID: "43e3d0d6-1a8e-4536-bc46-b5785124405a"). InnerVolumeSpecName "kube-api-access-fphlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:10.973237 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:10.973205    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43e3d0d6-1a8e-4536-bc46-b5785124405a-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:20:10.973237 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:10.973235    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43e3d0d6-1a8e-4536-bc46-b5785124405a-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:20:10.973501 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:10.973248    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fphlq\" (UniqueName: \"kubernetes.io/projected/43e3d0d6-1a8e-4536-bc46-b5785124405a-kube-api-access-fphlq\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:20:11.049628 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:11.049585    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b" podUID="6f480410-faab-4712-b042-4e02d5759d11" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:20:11.049822 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:11.049598    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr" podUID="82d7bfa3-6d24-4821-bde2-5272ab59b0c3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:20:11.387126 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:11.387085    4294 generic.go:356] "Generic (PLEG): container finished" podID="43e3d0d6-1a8e-4536-bc46-b5785124405a" containerID="43911fcc2bc6f85a5cab5a2d21c83dc1b186416ec975aaa39a5a3e07c584cfab" exitCode=0
Nov 03 00:20:11.387338 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:11.387135    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"
Nov 03 00:20:11.387338 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:11.387168    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9" event={"ID":"43e3d0d6-1a8e-4536-bc46-b5785124405a","Type":"ContainerDied","Data":"43911fcc2bc6f85a5cab5a2d21c83dc1b186416ec975aaa39a5a3e07c584cfab"}
Nov 03 00:20:11.387338 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:11.387210    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9" event={"ID":"43e3d0d6-1a8e-4536-bc46-b5785124405a","Type":"ContainerDied","Data":"f505f084b1b63470789380d2cb9ea0fded396d9df49c2b2bde1a043b2a2af07d"}
Nov 03 00:20:11.387338 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:11.387228    4294 scope.go:122] "RemoveContainer" containerID="43911fcc2bc6f85a5cab5a2d21c83dc1b186416ec975aaa39a5a3e07c584cfab"
Nov 03 00:20:11.398093 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:11.398066    4294 scope.go:122] "RemoveContainer" containerID="43911fcc2bc6f85a5cab5a2d21c83dc1b186416ec975aaa39a5a3e07c584cfab"
Nov 03 00:20:11.398379 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:20:11.398359    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43911fcc2bc6f85a5cab5a2d21c83dc1b186416ec975aaa39a5a3e07c584cfab\": container with ID starting with 43911fcc2bc6f85a5cab5a2d21c83dc1b186416ec975aaa39a5a3e07c584cfab not found: ID does not exist" containerID="43911fcc2bc6f85a5cab5a2d21c83dc1b186416ec975aaa39a5a3e07c584cfab"
Nov 03 00:20:11.398464 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:11.398386    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43911fcc2bc6f85a5cab5a2d21c83dc1b186416ec975aaa39a5a3e07c584cfab"} err="failed to get container status \"43911fcc2bc6f85a5cab5a2d21c83dc1b186416ec975aaa39a5a3e07c584cfab\": rpc error: code = NotFound desc = could not find container \"43911fcc2bc6f85a5cab5a2d21c83dc1b186416ec975aaa39a5a3e07c584cfab\": container with ID starting with 43911fcc2bc6f85a5cab5a2d21c83dc1b186416ec975aaa39a5a3e07c584cfab not found: ID does not exist"
Nov 03 00:20:11.544209 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:11.544173    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"]
Nov 03 00:20:11.591841 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:11.591809    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fb6e1d/simulator-w4-fb6e1d-kserve-5666bd6d6c-qtwx9"]
Nov 03 00:20:11.751596 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:20:11.751560    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:20:12.545780 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:12.545741    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43e3d0d6-1a8e-4536-bc46-b5785124405a" path="/var/lib/kubelet/pods/43e3d0d6-1a8e-4536-bc46-b5785124405a/volumes"
Nov 03 00:20:12.559579 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:12.559511    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:12.791004 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:12.790616    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:15.107022 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:15.106972    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:15.374162 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:15.374076    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:20.570066 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:20.570032    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:20.751509 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:20.751472    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:21.050596 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:21.050561    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b" podUID="6f480410-faab-4712-b042-4e02d5759d11" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:20:21.050788 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:21.050594    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr" podUID="82d7bfa3-6d24-4821-bde2-5272ab59b0c3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:20:24.888345 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.888267    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"]
Nov 03 00:20:24.888836 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.888816    4294 cpu_manager.go:414] "containerMap: removing container" podUID="723b2ff2-4870-4de1-b820-6813a12699b7" containerName="payload-processing"
Nov 03 00:20:24.888940 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.888838    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="723b2ff2-4870-4de1-b820-6813a12699b7" containerName="payload-processing"
Nov 03 00:20:24.888940 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.888860    4294 cpu_manager.go:414] "containerMap: removing container" podUID="4079e7c5-252d-40d6-ac37-67dde32884f6" containerName="payload-pre-processing"
Nov 03 00:20:24.888940 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.888870    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4079e7c5-252d-40d6-ac37-67dde32884f6" containerName="payload-pre-processing"
Nov 03 00:20:24.888940 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.888883    4294 cpu_manager.go:414] "containerMap: removing container" podUID="3c5057a9-8986-4bc3-8a01-2866a8aa09ca" containerName="payload-processing"
Nov 03 00:20:24.888940 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.888890    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c5057a9-8986-4bc3-8a01-2866a8aa09ca" containerName="payload-processing"
Nov 03 00:20:24.888940 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.888911    4294 cpu_manager.go:414] "containerMap: removing container" podUID="8868fdae-c6a7-4eb7-bef0-255f6586e34c" containerName="istio-proxy"
Nov 03 00:20:24.888940 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.888919    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8868fdae-c6a7-4eb7-bef0-255f6586e34c" containerName="istio-proxy"
Nov 03 00:20:24.888940 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.888938    4294 cpu_manager.go:414] "containerMap: removing container" podUID="705e82db-2105-4a14-8990-5f0b1322da1d" containerName="payload-pre-processing"
Nov 03 00:20:24.889425 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.888946    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705e82db-2105-4a14-8990-5f0b1322da1d" containerName="payload-pre-processing"
Nov 03 00:20:24.889425 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.888964    4294 cpu_manager.go:414] "containerMap: removing container" podUID="43e3d0d6-1a8e-4536-bc46-b5785124405a" containerName="main"
Nov 03 00:20:24.889425 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.888971    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e3d0d6-1a8e-4536-bc46-b5785124405a" containerName="main"
Nov 03 00:20:24.889425 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.888996    4294 cpu_manager.go:414] "containerMap: removing container" podUID="00c80137-cc6a-48e5-9e02-184ee47f56da" containerName="maas-api"
Nov 03 00:20:24.889425 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.889003    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c80137-cc6a-48e5-9e02-184ee47f56da" containerName="maas-api"
Nov 03 00:20:24.889425 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.889022    4294 cpu_manager.go:414] "containerMap: removing container" podUID="fba0cc63-6a7a-474c-b6e4-c411f912eb46" containerName="payload-pre-processing"
Nov 03 00:20:24.889425 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.889029    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba0cc63-6a7a-474c-b6e4-c411f912eb46" containerName="payload-pre-processing"
Nov 03 00:20:24.889425 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.889138    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba0cc63-6a7a-474c-b6e4-c411f912eb46" containerName="payload-pre-processing"
Nov 03 00:20:24.889425 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.889154    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="723b2ff2-4870-4de1-b820-6813a12699b7" containerName="payload-processing"
Nov 03 00:20:24.889425 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.889166    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4079e7c5-252d-40d6-ac37-67dde32884f6" containerName="payload-pre-processing"
Nov 03 00:20:24.889425 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.889177    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c5057a9-8986-4bc3-8a01-2866a8aa09ca" containerName="payload-processing"
Nov 03 00:20:24.889425 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.889188    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8868fdae-c6a7-4eb7-bef0-255f6586e34c" containerName="istio-proxy"
Nov 03 00:20:24.889425 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.889198    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="705e82db-2105-4a14-8990-5f0b1322da1d" containerName="payload-pre-processing"
Nov 03 00:20:24.889425 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.889210    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e3d0d6-1a8e-4536-bc46-b5785124405a" containerName="main"
Nov 03 00:20:24.889425 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.889219    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c80137-cc6a-48e5-9e02-184ee47f56da" containerName="maas-api"
Nov 03 00:20:24.893628 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.893609    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:24.943801 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.943777    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-fb6e1d-openshift-default-dockercfg-5lz5h\""
Nov 03 00:20:24.993347 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:24.993313    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"]
Nov 03 00:20:25.015025 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.014996    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.015202 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.015052    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.015202 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.015126    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.015202 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.015182    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.015329 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.015228    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.015329 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.015299    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsm5b\" (UniqueName: \"kubernetes.io/projected/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-kube-api-access-zsm5b\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.015329 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.015322    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.015442 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.015379    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.015442 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.015417    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.116095 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.116059    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.116095 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.116098    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.116309 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.116199    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.116309 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.116253    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.116411 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.116326    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsm5b\" (UniqueName: \"kubernetes.io/projected/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-kube-api-access-zsm5b\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.116411 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.116353    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.116411 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.116381    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.116571 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.116433    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.116571 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.116481    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.116713 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.116685    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-data\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.116803 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.116770    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-credential-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.116895 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.116877    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-workload-socket\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.117103 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.117081    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-workload-certs\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.117232 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.117208    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istiod-ca-cert\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.118681 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.118652    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-envoy\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.118807 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.118786    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-podinfo\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.171576 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.171517    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-token\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.171714 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.171628    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsm5b\" (UniqueName: \"kubernetes.io/projected/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-kube-api-access-zsm5b\") pod \"e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") " pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.206458 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.206423    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:25.616670 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.616635    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"]
Nov 03 00:20:25.653045 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:20:25.653014    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97ee0b5_11bf_4a79_a3e3_7a75b5c17ed7.slice/crio-4659f8a8b52accbbc633e53f32a9d35172c1099da4ae6cac04d415ccd89786dd WatchSource:0}: Error finding container 4659f8a8b52accbbc633e53f32a9d35172c1099da4ae6cac04d415ccd89786dd: Status 404 returned error can't find the container with id 4659f8a8b52accbbc633e53f32a9d35172c1099da4ae6cac04d415ccd89786dd
Nov 03 00:20:25.655210 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.655174    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:25.655306 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.655245    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:25.655306 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:25.655271    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:26.483002 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:26.482949    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w" event={"ID":"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7","Type":"ContainerStarted","Data":"28ce9419f7a812351ea95b0849214ca2cd45bb65df5637e39446ce22e7cd80c6"}
Nov 03 00:20:26.483457 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:26.483019    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w" event={"ID":"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7","Type":"ContainerStarted","Data":"4659f8a8b52accbbc633e53f32a9d35172c1099da4ae6cac04d415ccd89786dd"}
Nov 03 00:20:26.596871 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:26.596813    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w" podStartSLOduration=2.5967972230000003 podStartE2EDuration="2.596797223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:26.595512922 +0000 UTC m=+5266.163867711" watchObservedRunningTime="2026-11-03 00:20:26.596797223 +0000 UTC m=+5266.165152015"
Nov 03 00:20:27.207441 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:27.207401    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:27.209115 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:27.209088    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 00:20:27.209242 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:27.209140    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w" podUID="f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 00:20:28.207234 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:28.207196    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 00:20:28.207771 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:28.207261    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w" podUID="f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 00:20:29.207759 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.207723    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 00:20:29.208152 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.207787    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w" podUID="f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 00:20:29.302347 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.302319    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b"
Nov 03 00:20:29.443863 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.443840    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr"
Nov 03 00:20:29.481594 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.481555    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f480410-faab-4712-b042-4e02d5759d11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f480410-faab-4712-b042-4e02d5759d11-plugins-config-volume\") pod \"6f480410-faab-4712-b042-4e02d5759d11\" (UID: \"6f480410-faab-4712-b042-4e02d5759d11\") "
Nov 03 00:20:29.481594 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.481600    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f480410-faab-4712-b042-4e02d5759d11-kube-api-access-k527k\" (UniqueName: \"kubernetes.io/projected/6f480410-faab-4712-b042-4e02d5759d11-kube-api-access-k527k\") pod \"6f480410-faab-4712-b042-4e02d5759d11\" (UID: \"6f480410-faab-4712-b042-4e02d5759d11\") "
Nov 03 00:20:29.481903 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.481879    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f480410-faab-4712-b042-4e02d5759d11-plugins-config-volume" pod "6f480410-faab-4712-b042-4e02d5759d11" (UID: "6f480410-faab-4712-b042-4e02d5759d11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:20:29.483839 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.483814    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f480410-faab-4712-b042-4e02d5759d11-kube-api-access-k527k" pod "6f480410-faab-4712-b042-4e02d5759d11" (UID: "6f480410-faab-4712-b042-4e02d5759d11"). InnerVolumeSpecName "kube-api-access-k527k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:29.498414 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.498382    4294 generic.go:356] "Generic (PLEG): container finished" podID="82d7bfa3-6d24-4821-bde2-5272ab59b0c3" containerID="e80be45d5d272679e8ac8c1712067ba42a7c3ab8a6ed4ce73ec212d018b37751" exitCode=0
Nov 03 00:20:29.498589 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.498424    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr"
Nov 03 00:20:29.498589 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.498457    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr" event={"ID":"82d7bfa3-6d24-4821-bde2-5272ab59b0c3","Type":"ContainerDied","Data":"e80be45d5d272679e8ac8c1712067ba42a7c3ab8a6ed4ce73ec212d018b37751"}
Nov 03 00:20:29.498589 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.498508    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr" event={"ID":"82d7bfa3-6d24-4821-bde2-5272ab59b0c3","Type":"ContainerDied","Data":"649ce0f273e4f3ad2b0fda57f8ada58cd393ee6d6f4f858864583d4662e7d888"}
Nov 03 00:20:29.498589 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.498529    4294 scope.go:122] "RemoveContainer" containerID="e80be45d5d272679e8ac8c1712067ba42a7c3ab8a6ed4ce73ec212d018b37751"
Nov 03 00:20:29.499842 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.499820    4294 generic.go:356] "Generic (PLEG): container finished" podID="6f480410-faab-4712-b042-4e02d5759d11" containerID="65abc4b4e8e5fb7a304380ca64c9681f8d95670734fe8e711b4bc062ae61a9fe" exitCode=0
Nov 03 00:20:29.499944 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.499849    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b" event={"ID":"6f480410-faab-4712-b042-4e02d5759d11","Type":"ContainerDied","Data":"65abc4b4e8e5fb7a304380ca64c9681f8d95670734fe8e711b4bc062ae61a9fe"}
Nov 03 00:20:29.499944 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.499864    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b"
Nov 03 00:20:29.499944 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.499889    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b" event={"ID":"6f480410-faab-4712-b042-4e02d5759d11","Type":"ContainerDied","Data":"4deda747dcc4e119e42cbe78821f55c99f31d845c19dad31506c075622101282"}
Nov 03 00:20:29.508475 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.508454    4294 scope.go:122] "RemoveContainer" containerID="e80be45d5d272679e8ac8c1712067ba42a7c3ab8a6ed4ce73ec212d018b37751"
Nov 03 00:20:29.508862 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:20:29.508832    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80be45d5d272679e8ac8c1712067ba42a7c3ab8a6ed4ce73ec212d018b37751\": container with ID starting with e80be45d5d272679e8ac8c1712067ba42a7c3ab8a6ed4ce73ec212d018b37751 not found: ID does not exist" containerID="e80be45d5d272679e8ac8c1712067ba42a7c3ab8a6ed4ce73ec212d018b37751"
Nov 03 00:20:29.508947 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.508874    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80be45d5d272679e8ac8c1712067ba42a7c3ab8a6ed4ce73ec212d018b37751"} err="failed to get container status \"e80be45d5d272679e8ac8c1712067ba42a7c3ab8a6ed4ce73ec212d018b37751\": rpc error: code = NotFound desc = could not find container \"e80be45d5d272679e8ac8c1712067ba42a7c3ab8a6ed4ce73ec212d018b37751\": container with ID starting with e80be45d5d272679e8ac8c1712067ba42a7c3ab8a6ed4ce73ec212d018b37751 not found: ID does not exist"
Nov 03 00:20:29.508947 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.508899    4294 scope.go:122] "RemoveContainer" containerID="65abc4b4e8e5fb7a304380ca64c9681f8d95670734fe8e711b4bc062ae61a9fe"
Nov 03 00:20:29.516699 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.516680    4294 scope.go:122] "RemoveContainer" containerID="65abc4b4e8e5fb7a304380ca64c9681f8d95670734fe8e711b4bc062ae61a9fe"
Nov 03 00:20:29.516960 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:20:29.516941    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65abc4b4e8e5fb7a304380ca64c9681f8d95670734fe8e711b4bc062ae61a9fe\": container with ID starting with 65abc4b4e8e5fb7a304380ca64c9681f8d95670734fe8e711b4bc062ae61a9fe not found: ID does not exist" containerID="65abc4b4e8e5fb7a304380ca64c9681f8d95670734fe8e711b4bc062ae61a9fe"
Nov 03 00:20:29.517053 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.516973    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65abc4b4e8e5fb7a304380ca64c9681f8d95670734fe8e711b4bc062ae61a9fe"} err="failed to get container status \"65abc4b4e8e5fb7a304380ca64c9681f8d95670734fe8e711b4bc062ae61a9fe\": rpc error: code = NotFound desc = could not find container \"65abc4b4e8e5fb7a304380ca64c9681f8d95670734fe8e711b4bc062ae61a9fe\": container with ID starting with 65abc4b4e8e5fb7a304380ca64c9681f8d95670734fe8e711b4bc062ae61a9fe not found: ID does not exist"
Nov 03 00:20:29.582187 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.582093    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82d7bfa3-6d24-4821-bde2-5272ab59b0c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d7bfa3-6d24-4821-bde2-5272ab59b0c3-plugins-config-volume\") pod \"82d7bfa3-6d24-4821-bde2-5272ab59b0c3\" (UID: \"82d7bfa3-6d24-4821-bde2-5272ab59b0c3\") "
Nov 03 00:20:29.582582 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.582554    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d7bfa3-6d24-4821-bde2-5272ab59b0c3-plugins-config-volume" pod "82d7bfa3-6d24-4821-bde2-5272ab59b0c3" (UID: "82d7bfa3-6d24-4821-bde2-5272ab59b0c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:20:29.582656 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.582619    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82d7bfa3-6d24-4821-bde2-5272ab59b0c3-kube-api-access-9pcv4\" (UniqueName: \"kubernetes.io/projected/82d7bfa3-6d24-4821-bde2-5272ab59b0c3-kube-api-access-9pcv4\") pod \"82d7bfa3-6d24-4821-bde2-5272ab59b0c3\" (UID: \"82d7bfa3-6d24-4821-bde2-5272ab59b0c3\") "
Nov 03 00:20:29.583067 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.583047    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f480410-faab-4712-b042-4e02d5759d11-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:20:29.583067 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.583066    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k527k\" (UniqueName: \"kubernetes.io/projected/6f480410-faab-4712-b042-4e02d5759d11-kube-api-access-k527k\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:20:29.583210 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.583075    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d7bfa3-6d24-4821-bde2-5272ab59b0c3-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:20:29.584849 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.584817    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d7bfa3-6d24-4821-bde2-5272ab59b0c3-kube-api-access-9pcv4" pod "82d7bfa3-6d24-4821-bde2-5272ab59b0c3" (UID: "82d7bfa3-6d24-4821-bde2-5272ab59b0c3"). InnerVolumeSpecName "kube-api-access-9pcv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:29.684665 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.684626    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pcv4\" (UniqueName: \"kubernetes.io/projected/82d7bfa3-6d24-4821-bde2-5272ab59b0c3-kube-api-access-9pcv4\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:20:29.733513 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.733479    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b"]
Nov 03 00:20:29.738499 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.738463    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-fb6e1d-79c65f76dc-cx89b"]
Nov 03 00:20:29.922096 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.922009    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr"]
Nov 03 00:20:29.928095 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:29.928067    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fb6e1d-8597b658f9-6tvpr"]
Nov 03 00:20:30.207710 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:30.207627    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 00:20:30.207710 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:30.207680    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w" podUID="f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 00:20:30.546295 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:30.546264    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f480410-faab-4712-b042-4e02d5759d11" path="/var/lib/kubelet/pods/6f480410-faab-4712-b042-4e02d5759d11/volumes"
Nov 03 00:20:30.546645 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:30.546633    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82d7bfa3-6d24-4821-bde2-5272ab59b0c3" path="/var/lib/kubelet/pods/82d7bfa3-6d24-4821-bde2-5272ab59b0c3/volumes"
Nov 03 00:20:31.207665 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:31.207622    4294 patch_prober.go:29] interesting pod/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Nov 03 00:20:31.207851 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:31.207708    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w" podUID="f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Nov 03 00:20:32.210857 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:32.210828    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:32.211266 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:32.211086    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:32.211843 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:32.211826    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:32.828132 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:32.828071    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:33.204253 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:33.204173    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:33.377889 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:33.377859    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:33.543613 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:33.543578    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:35.069865 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:35.069810    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:35.513040 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:35.513009    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"]
Nov 03 00:20:35.513215 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:35.513191    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w" podUID="f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7" containerName="istio-proxy" containerID="cri-o://28ce9419f7a812351ea95b0849214ca2cd45bb65df5637e39446ce22e7cd80c6" gracePeriod=30
Nov 03 00:20:35.555965 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:35.555856    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:36.490038 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:36.490006    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-575d76f44f-dlvmr_0cb7b02e-68fb-4619-a81c-42bcb8113a5c/authorino/0.log"
Nov 03 00:20:38.971140 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:38.971109    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:39.142482 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:39.142444    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:39.847348 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:39.847307    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-669b4bd578-5cj8x_ac0193db-08d3-4abb-9a82-c2199ba08ccf/manager/0.log"
Nov 03 00:20:39.856492 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:39.856465    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:40.028679 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.028651    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:40.751233 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.751209    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:40.790788 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.790730    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-workload-socket\") pod \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") "
Nov 03 00:20:40.790788 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.790769    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-kube-api-access-zsm5b\" (UniqueName: \"kubernetes.io/projected/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-kube-api-access-zsm5b\") pod \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") "
Nov 03 00:20:40.791043 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.790815    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-data\") pod \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") "
Nov 03 00:20:40.791043 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.790850    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-envoy\") pod \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") "
Nov 03 00:20:40.791043 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.790881    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-workload-certs\") pod \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") "
Nov 03 00:20:40.791043 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.790915    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-credential-socket\") pod \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") "
Nov 03 00:20:40.791043 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.790950    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-podinfo\") pod \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") "
Nov 03 00:20:40.791043 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.791003    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-token\" (UniqueName: \"kubernetes.io/projected/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-token\") pod \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") "
Nov 03 00:20:40.791352 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.791043    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-workload-socket" pod "f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7" (UID: "f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:20:40.791352 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.791042    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istiod-ca-cert\") pod \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\" (UID: \"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7\") "
Nov 03 00:20:40.791352 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.791274    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-workload-certs" pod "f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7" (UID: "f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:20:40.791538 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.791404    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-data" pod "f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7" (UID: "f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:20:40.791538 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.791435    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-credential-socket" pod "f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7" (UID: "f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:20:40.791538 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.791505    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istiod-ca-cert" pod "f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7" (UID: "f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:20:40.791712 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.791568    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:20:40.791712 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.791586    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:20:40.791712 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.791600    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:20:40.791712 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.791613    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:20:40.793228 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.793199    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-token" pod "f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7" (UID: "f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:40.793558 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.793531    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-kube-api-access-zsm5b" pod "f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7" (UID: "f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7"). InnerVolumeSpecName "kube-api-access-zsm5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:40.793671 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.793557    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-podinfo" pod "f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7" (UID: "f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:20:40.793671 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.793612    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-envoy" pod "f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7" (UID: "f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:20:40.892381 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.892346    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:20:40.892381 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.892375    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:20:40.892381 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.892385    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:20:40.892622 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.892393    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsm5b\" (UniqueName: \"kubernetes.io/projected/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-kube-api-access-zsm5b\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:20:40.892622 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:40.892403    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:20:41.563393 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:41.563357    4294 generic.go:356] "Generic (PLEG): container finished" podID="f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7" containerID="28ce9419f7a812351ea95b0849214ca2cd45bb65df5637e39446ce22e7cd80c6" exitCode=0
Nov 03 00:20:41.563904 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:41.563409    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"
Nov 03 00:20:41.563904 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:41.563437    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w" event={"ID":"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7","Type":"ContainerDied","Data":"28ce9419f7a812351ea95b0849214ca2cd45bb65df5637e39446ce22e7cd80c6"}
Nov 03 00:20:41.563904 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:41.563489    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w" event={"ID":"f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7","Type":"ContainerDied","Data":"4659f8a8b52accbbc633e53f32a9d35172c1099da4ae6cac04d415ccd89786dd"}
Nov 03 00:20:41.563904 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:41.563511    4294 scope.go:122] "RemoveContainer" containerID="28ce9419f7a812351ea95b0849214ca2cd45bb65df5637e39446ce22e7cd80c6"
Nov 03 00:20:41.572843 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:41.572824    4294 scope.go:122] "RemoveContainer" containerID="28ce9419f7a812351ea95b0849214ca2cd45bb65df5637e39446ce22e7cd80c6"
Nov 03 00:20:41.573129 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:20:41.573109    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ce9419f7a812351ea95b0849214ca2cd45bb65df5637e39446ce22e7cd80c6\": container with ID starting with 28ce9419f7a812351ea95b0849214ca2cd45bb65df5637e39446ce22e7cd80c6 not found: ID does not exist" containerID="28ce9419f7a812351ea95b0849214ca2cd45bb65df5637e39446ce22e7cd80c6"
Nov 03 00:20:41.573199 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:41.573135    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ce9419f7a812351ea95b0849214ca2cd45bb65df5637e39446ce22e7cd80c6"} err="failed to get container status \"28ce9419f7a812351ea95b0849214ca2cd45bb65df5637e39446ce22e7cd80c6\": rpc error: code = NotFound desc = could not find container \"28ce9419f7a812351ea95b0849214ca2cd45bb65df5637e39446ce22e7cd80c6\": container with ID starting with 28ce9419f7a812351ea95b0849214ca2cd45bb65df5637e39446ce22e7cd80c6 not found: ID does not exist"
Nov 03 00:20:41.654520 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:41.654491    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4njxm_5b22dd01-17ba-4504-9208-7e3f35ff268b/manager/0.log"
Nov 03 00:20:41.749030 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:20:41.748975    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:20:41.769341 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:41.769309    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"]
Nov 03 00:20:41.817657 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:41.817582    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-fb6e1d-openshift-default-745d8f469-2h89w"]
Nov 03 00:20:42.546127 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:42.546098    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7" path="/var/lib/kubelet/pods/f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7/volumes"
Nov 03 00:20:46.505120 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:46.505083    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:47.101811 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:47.101776    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:47.346757 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:47.346721    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:50.938057 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:50.938019    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:51.123674 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:51.123627    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:51.444079 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:51.444044    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:54.070642 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.070610    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:54.249635 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.249599    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:54.442923 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.442836    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9"]
Nov 03 00:20:54.443326 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.443306    4294 cpu_manager.go:414] "containerMap: removing container" podUID="f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7" containerName="istio-proxy"
Nov 03 00:20:54.443428 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.443328    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7" containerName="istio-proxy"
Nov 03 00:20:54.443428 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.443359    4294 cpu_manager.go:414] "containerMap: removing container" podUID="82d7bfa3-6d24-4821-bde2-5272ab59b0c3" containerName="payload-pre-processing"
Nov 03 00:20:54.443428 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.443368    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82d7bfa3-6d24-4821-bde2-5272ab59b0c3" containerName="payload-pre-processing"
Nov 03 00:20:54.443428 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.443400    4294 cpu_manager.go:414] "containerMap: removing container" podUID="6f480410-faab-4712-b042-4e02d5759d11" containerName="payload-processing"
Nov 03 00:20:54.443428 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.443408    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f480410-faab-4712-b042-4e02d5759d11" containerName="payload-processing"
Nov 03 00:20:54.443697 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.443506    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97ee0b5-11bf-4a79-a3e3-7a75b5c17ed7" containerName="istio-proxy"
Nov 03 00:20:54.443697 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.443522    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="82d7bfa3-6d24-4821-bde2-5272ab59b0c3" containerName="payload-pre-processing"
Nov 03 00:20:54.443697 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.443537    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f480410-faab-4712-b042-4e02d5759d11" containerName="payload-processing"
Nov 03 00:20:54.448115 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.448095    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9"
Nov 03 00:20:54.499607 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.499582    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9"]
Nov 03 00:20:54.516894 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.516869    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w969\" (UniqueName: \"kubernetes.io/projected/b8ce7481-6c79-4c18-a7a8-67c0bb763b27-kube-api-access-8w969\") pod \"payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9\" (UID: \"b8ce7481-6c79-4c18-a7a8-67c0bb763b27\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9"
Nov 03 00:20:54.517037 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.517021    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ce7481-6c79-4c18-a7a8-67c0bb763b27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9\" (UID: \"b8ce7481-6c79-4c18-a7a8-67c0bb763b27\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9"
Nov 03 00:20:54.545016 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.544974    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a2a3480b\""
Nov 03 00:20:54.545016 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.545011    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a2a3480b-dockercfg-kpf9f\""
Nov 03 00:20:54.599350 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.599312    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml"]
Nov 03 00:20:54.602766 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.602750    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml"
Nov 03 00:20:54.617739 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.617718    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ce7481-6c79-4c18-a7a8-67c0bb763b27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9\" (UID: \"b8ce7481-6c79-4c18-a7a8-67c0bb763b27\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9"
Nov 03 00:20:54.617859 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.617761    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w969\" (UniqueName: \"kubernetes.io/projected/b8ce7481-6c79-4c18-a7a8-67c0bb763b27-kube-api-access-8w969\") pod \"payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9\" (UID: \"b8ce7481-6c79-4c18-a7a8-67c0bb763b27\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9"
Nov 03 00:20:54.618389 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.618370    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ce7481-6c79-4c18-a7a8-67c0bb763b27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9\" (UID: \"b8ce7481-6c79-4c18-a7a8-67c0bb763b27\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9"
Nov 03 00:20:54.657180 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.657153    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml"]
Nov 03 00:20:54.672561 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.672532    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w969\" (UniqueName: \"kubernetes.io/projected/b8ce7481-6c79-4c18-a7a8-67c0bb763b27-kube-api-access-8w969\") pod \"payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9\" (UID: \"b8ce7481-6c79-4c18-a7a8-67c0bb763b27\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9"
Nov 03 00:20:54.718480 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.718404    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e18b4ca-516d-4219-86f2-1ff536732102-plugins-config-volume\") pod \"payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml\" (UID: \"2e18b4ca-516d-4219-86f2-1ff536732102\") " pod="openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml"
Nov 03 00:20:54.718480 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.718445    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk2v2\" (UniqueName: \"kubernetes.io/projected/2e18b4ca-516d-4219-86f2-1ff536732102-kube-api-access-vk2v2\") pod \"payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml\" (UID: \"2e18b4ca-516d-4219-86f2-1ff536732102\") " pod="openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml"
Nov 03 00:20:54.761345 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.761317    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9"
Nov 03 00:20:54.819738 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.819705    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e18b4ca-516d-4219-86f2-1ff536732102-plugins-config-volume\") pod \"payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml\" (UID: \"2e18b4ca-516d-4219-86f2-1ff536732102\") " pod="openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml"
Nov 03 00:20:54.819873 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.819774    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk2v2\" (UniqueName: \"kubernetes.io/projected/2e18b4ca-516d-4219-86f2-1ff536732102-kube-api-access-vk2v2\") pod \"payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml\" (UID: \"2e18b4ca-516d-4219-86f2-1ff536732102\") " pod="openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml"
Nov 03 00:20:54.820473 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.820450    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e18b4ca-516d-4219-86f2-1ff536732102-plugins-config-volume\") pod \"payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml\" (UID: \"2e18b4ca-516d-4219-86f2-1ff536732102\") " pod="openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml"
Nov 03 00:20:54.874571 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.874539    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk2v2\" (UniqueName: \"kubernetes.io/projected/2e18b4ca-516d-4219-86f2-1ff536732102-kube-api-access-vk2v2\") pod \"payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml\" (UID: \"2e18b4ca-516d-4219-86f2-1ff536732102\") " pod="openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml"
Nov 03 00:20:54.915537 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:54.915507    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml"
Nov 03 00:20:55.125814 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:55.125785    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9"]
Nov 03 00:20:55.126872 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:20:55.126845    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ce7481_6c79_4c18_a7a8_67c0bb763b27.slice/crio-fd55a2ad0e9d9a487200c9b02e022c8df4380b641448af958e50b1108ba7352f WatchSource:0}: Error finding container fd55a2ad0e9d9a487200c9b02e022c8df4380b641448af958e50b1108ba7352f: Status 404 returned error can't find the container with id fd55a2ad0e9d9a487200c9b02e022c8df4380b641448af958e50b1108ba7352f
Nov 03 00:20:55.288302 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:55.288277    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml"]
Nov 03 00:20:55.289599 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:20:55.289567    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e18b4ca_516d_4219_86f2_1ff536732102.slice/crio-53f0886bbec9d89c0a15a92e6a8f0822d3322441db2b2cba112547d7fb91f01e WatchSource:0}: Error finding container 53f0886bbec9d89c0a15a92e6a8f0822d3322441db2b2cba112547d7fb91f01e: Status 404 returned error can't find the container with id 53f0886bbec9d89c0a15a92e6a8f0822d3322441db2b2cba112547d7fb91f01e
Nov 03 00:20:55.626901 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:55.626806    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml" event={"ID":"2e18b4ca-516d-4219-86f2-1ff536732102","Type":"ContainerStarted","Data":"3f03ad4f563d09f01f9bb91d5536b60adc64343740d946586198b494fdd100b4"}
Nov 03 00:20:55.627166 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:55.626899    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml" event={"ID":"2e18b4ca-516d-4219-86f2-1ff536732102","Type":"ContainerStarted","Data":"53f0886bbec9d89c0a15a92e6a8f0822d3322441db2b2cba112547d7fb91f01e"}
Nov 03 00:20:55.627166 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:55.627096    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml"
Nov 03 00:20:55.628444 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:55.628422    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9" event={"ID":"b8ce7481-6c79-4c18-a7a8-67c0bb763b27","Type":"ContainerStarted","Data":"e1ca192ff9d854cf375415aae5cf9df39ef0a68f97cbd9c72204c42a7679a1ca"}
Nov 03 00:20:55.628537 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:55.628462    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9" event={"ID":"b8ce7481-6c79-4c18-a7a8-67c0bb763b27","Type":"ContainerStarted","Data":"fd55a2ad0e9d9a487200c9b02e022c8df4380b641448af958e50b1108ba7352f"}
Nov 03 00:20:55.628537 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:55.628484    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9"
Nov 03 00:20:55.737080 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:55.737022    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml" podStartSLOduration=1.737007116 podStartE2EDuration="1.737007116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:55.736264138 +0000 UTC m=+5295.304618925" watchObservedRunningTime="2026-11-03 00:20:55.737007116 +0000 UTC m=+5295.305361904"
Nov 03 00:20:55.852260 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:55.852212    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9" podStartSLOduration=1.8521978 podStartE2EDuration="1.8521978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:55.851321511 +0000 UTC m=+5295.419676299" watchObservedRunningTime="2026-11-03 00:20:55.8521978 +0000 UTC m=+5295.420552631"
Nov 03 00:20:56.986512 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:56.986481    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:57.153093 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:57.153058    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:57.319617 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:20:57.319587    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:01.752559 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:01.752526    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:06.636436 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:06.636410    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9"
Nov 03 00:21:06.636851 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:06.636469    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml"
Nov 03 00:21:06.745854 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:06.745811    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:11.330318 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:11.330286    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:11.750037 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:21:11.750004    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:21:16.238174 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:16.238139    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:21.439035 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:21.438952    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:26.146897 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:26.146862    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:28.663293 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:28.663257    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml"]
Nov 03 00:21:28.762596 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:28.762563    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9"]
Nov 03 00:21:29.031809 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:29.031771    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:29.200076 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:29.200039    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:30.739409 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:30.739373    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:33.664014 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:33.663946    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml" podUID="2e18b4ca-516d-4219-86f2-1ff536732102" containerName="payload-processing" containerID="cri-o://3f03ad4f563d09f01f9bb91d5536b60adc64343740d946586198b494fdd100b4" gracePeriod=25
Nov 03 00:21:33.763626 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:33.763584    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9" podUID="b8ce7481-6c79-4c18-a7a8-67c0bb763b27" containerName="payload-pre-processing" containerID="cri-o://e1ca192ff9d854cf375415aae5cf9df39ef0a68f97cbd9c72204c42a7679a1ca" gracePeriod=25
Nov 03 00:21:33.814750 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:33.814714    4294 generic.go:356] "Generic (PLEG): container finished" podID="2e18b4ca-516d-4219-86f2-1ff536732102" containerID="3f03ad4f563d09f01f9bb91d5536b60adc64343740d946586198b494fdd100b4" exitCode=0
Nov 03 00:21:33.814951 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:33.814787    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml" event={"ID":"2e18b4ca-516d-4219-86f2-1ff536732102","Type":"ContainerDied","Data":"3f03ad4f563d09f01f9bb91d5536b60adc64343740d946586198b494fdd100b4"}
Nov 03 00:21:33.913519 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:33.913490    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml"
Nov 03 00:21:34.010428 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.010404    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e18b4ca-516d-4219-86f2-1ff536732102-kube-api-access-vk2v2\" (UniqueName: \"kubernetes.io/projected/2e18b4ca-516d-4219-86f2-1ff536732102-kube-api-access-vk2v2\") pod \"2e18b4ca-516d-4219-86f2-1ff536732102\" (UID: \"2e18b4ca-516d-4219-86f2-1ff536732102\") "
Nov 03 00:21:34.010601 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.010468    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e18b4ca-516d-4219-86f2-1ff536732102-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e18b4ca-516d-4219-86f2-1ff536732102-plugins-config-volume\") pod \"2e18b4ca-516d-4219-86f2-1ff536732102\" (UID: \"2e18b4ca-516d-4219-86f2-1ff536732102\") "
Nov 03 00:21:34.010808 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.010786    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e18b4ca-516d-4219-86f2-1ff536732102-plugins-config-volume" pod "2e18b4ca-516d-4219-86f2-1ff536732102" (UID: "2e18b4ca-516d-4219-86f2-1ff536732102"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:21:34.012484 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.012462    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e18b4ca-516d-4219-86f2-1ff536732102-kube-api-access-vk2v2" pod "2e18b4ca-516d-4219-86f2-1ff536732102" (UID: "2e18b4ca-516d-4219-86f2-1ff536732102"). InnerVolumeSpecName "kube-api-access-vk2v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:21:34.012697 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.012680    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9"
Nov 03 00:21:34.111322 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.111283    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8ce7481-6c79-4c18-a7a8-67c0bb763b27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ce7481-6c79-4c18-a7a8-67c0bb763b27-plugins-config-volume\") pod \"b8ce7481-6c79-4c18-a7a8-67c0bb763b27\" (UID: \"b8ce7481-6c79-4c18-a7a8-67c0bb763b27\") "
Nov 03 00:21:34.111322 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.111334    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ce7481-6c79-4c18-a7a8-67c0bb763b27-kube-api-access-8w969\" (UniqueName: \"kubernetes.io/projected/b8ce7481-6c79-4c18-a7a8-67c0bb763b27-kube-api-access-8w969\") pod \"b8ce7481-6c79-4c18-a7a8-67c0bb763b27\" (UID: \"b8ce7481-6c79-4c18-a7a8-67c0bb763b27\") "
Nov 03 00:21:34.111598 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.111492    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk2v2\" (UniqueName: \"kubernetes.io/projected/2e18b4ca-516d-4219-86f2-1ff536732102-kube-api-access-vk2v2\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:21:34.111598 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.111504    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e18b4ca-516d-4219-86f2-1ff536732102-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:21:34.111777 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.111752    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ce7481-6c79-4c18-a7a8-67c0bb763b27-plugins-config-volume" pod "b8ce7481-6c79-4c18-a7a8-67c0bb763b27" (UID: "b8ce7481-6c79-4c18-a7a8-67c0bb763b27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:21:34.113490 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.113470    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ce7481-6c79-4c18-a7a8-67c0bb763b27-kube-api-access-8w969" pod "b8ce7481-6c79-4c18-a7a8-67c0bb763b27" (UID: "b8ce7481-6c79-4c18-a7a8-67c0bb763b27"). InnerVolumeSpecName "kube-api-access-8w969". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:21:34.212475 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.212385    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ce7481-6c79-4c18-a7a8-67c0bb763b27-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:21:34.212475 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.212418    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w969\" (UniqueName: \"kubernetes.io/projected/b8ce7481-6c79-4c18-a7a8-67c0bb763b27-kube-api-access-8w969\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:21:34.821101 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.821020    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml"
Nov 03 00:21:34.821544 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.821018    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml" event={"ID":"2e18b4ca-516d-4219-86f2-1ff536732102","Type":"ContainerDied","Data":"53f0886bbec9d89c0a15a92e6a8f0822d3322441db2b2cba112547d7fb91f01e"}
Nov 03 00:21:34.821544 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.821195    4294 scope.go:122] "RemoveContainer" containerID="3f03ad4f563d09f01f9bb91d5536b60adc64343740d946586198b494fdd100b4"
Nov 03 00:21:34.822466 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.822359    4294 generic.go:356] "Generic (PLEG): container finished" podID="b8ce7481-6c79-4c18-a7a8-67c0bb763b27" containerID="e1ca192ff9d854cf375415aae5cf9df39ef0a68f97cbd9c72204c42a7679a1ca" exitCode=0
Nov 03 00:21:34.822466 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.822402    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9"
Nov 03 00:21:34.822466 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.822420    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9" event={"ID":"b8ce7481-6c79-4c18-a7a8-67c0bb763b27","Type":"ContainerDied","Data":"e1ca192ff9d854cf375415aae5cf9df39ef0a68f97cbd9c72204c42a7679a1ca"}
Nov 03 00:21:34.822680 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.822473    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9" event={"ID":"b8ce7481-6c79-4c18-a7a8-67c0bb763b27","Type":"ContainerDied","Data":"fd55a2ad0e9d9a487200c9b02e022c8df4380b641448af958e50b1108ba7352f"}
Nov 03 00:21:34.830053 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.830033    4294 scope.go:122] "RemoveContainer" containerID="e1ca192ff9d854cf375415aae5cf9df39ef0a68f97cbd9c72204c42a7679a1ca"
Nov 03 00:21:34.838156 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.838137    4294 scope.go:122] "RemoveContainer" containerID="e1ca192ff9d854cf375415aae5cf9df39ef0a68f97cbd9c72204c42a7679a1ca"
Nov 03 00:21:34.838437 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:21:34.838415    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ca192ff9d854cf375415aae5cf9df39ef0a68f97cbd9c72204c42a7679a1ca\": container with ID starting with e1ca192ff9d854cf375415aae5cf9df39ef0a68f97cbd9c72204c42a7679a1ca not found: ID does not exist" containerID="e1ca192ff9d854cf375415aae5cf9df39ef0a68f97cbd9c72204c42a7679a1ca"
Nov 03 00:21:34.838502 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.838446    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ca192ff9d854cf375415aae5cf9df39ef0a68f97cbd9c72204c42a7679a1ca"} err="failed to get container status \"e1ca192ff9d854cf375415aae5cf9df39ef0a68f97cbd9c72204c42a7679a1ca\": rpc error: code = NotFound desc = could not find container \"e1ca192ff9d854cf375415aae5cf9df39ef0a68f97cbd9c72204c42a7679a1ca\": container with ID starting with e1ca192ff9d854cf375415aae5cf9df39ef0a68f97cbd9c72204c42a7679a1ca not found: ID does not exist"
Nov 03 00:21:34.848094 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.848072    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:34.980636 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:34.980598    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9"]
Nov 03 00:21:35.027849 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:35.027816    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a2a3480b-5947dc8d9f-kw4s9"]
Nov 03 00:21:35.090363 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:35.090277    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml"]
Nov 03 00:21:35.094766 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:35.094741    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2a3480b-86cf4d855c-jkzml"]
Nov 03 00:21:36.285234 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:36.285199    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:36.537949 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:36.537869    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:36.546235 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:36.546207    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e18b4ca-516d-4219-86f2-1ff536732102" path="/var/lib/kubelet/pods/2e18b4ca-516d-4219-86f2-1ff536732102/volumes"
Nov 03 00:21:36.546597 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:36.546585    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ce7481-6c79-4c18-a7a8-67c0bb763b27" path="/var/lib/kubelet/pods/b8ce7481-6c79-4c18-a7a8-67c0bb763b27/volumes"
Nov 03 00:21:38.205404 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:38.205370    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:41.749666 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:21:41.749630    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:21:51.764887 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:21:51.764847    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:11.748947 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:22:11.748911    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:22:18.206931 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:22:18.206892    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:20.539273 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:22:20.539240    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:22.670339 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:22:22.670309    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:41.748188 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:22:41.748153    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:22:52.120162 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:22:52.120121    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:02.551316 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:02.551284    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:04.557656 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:04.557620    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:11.748335 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:23:11.748297    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:23:18.888466 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:18.888433    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"]
Nov 03 00:23:18.889022 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:18.888809    4294 cpu_manager.go:414] "containerMap: removing container" podUID="b8ce7481-6c79-4c18-a7a8-67c0bb763b27" containerName="payload-pre-processing"
Nov 03 00:23:18.889022 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:18.888821    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ce7481-6c79-4c18-a7a8-67c0bb763b27" containerName="payload-pre-processing"
Nov 03 00:23:18.889022 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:18.888834    4294 cpu_manager.go:414] "containerMap: removing container" podUID="2e18b4ca-516d-4219-86f2-1ff536732102" containerName="payload-processing"
Nov 03 00:23:18.889022 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:18.888839    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e18b4ca-516d-4219-86f2-1ff536732102" containerName="payload-processing"
Nov 03 00:23:18.889022 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:18.888917    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e18b4ca-516d-4219-86f2-1ff536732102" containerName="payload-processing"
Nov 03 00:23:18.889022 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:18.888934    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ce7481-6c79-4c18-a7a8-67c0bb763b27" containerName="payload-pre-processing"
Nov 03 00:23:18.891957 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:18.891936    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:18.936218 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:18.936191    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"]
Nov 03 00:23:18.939473 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:18.939458    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:18.986585 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:18.986553    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"]
Nov 03 00:23:19.033222 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.033191    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"]
Nov 03 00:23:19.082876 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.082828    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9b2053-3218-46fb-bfd7-b193dccc8109-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.082876 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.082878    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/096035a0-30e5-4124-ad5b-8bfe177aebf1-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.083179 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.083001    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/096035a0-30e5-4124-ad5b-8bfe177aebf1-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.083179 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.083054    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9b2053-3218-46fb-bfd7-b193dccc8109-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.083179 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.083122    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9b2053-3218-46fb-bfd7-b193dccc8109-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.083179 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.083150    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/096035a0-30e5-4124-ad5b-8bfe177aebf1-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.083450 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.083222    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9b2053-3218-46fb-bfd7-b193dccc8109-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.083450 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.083249    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9b2053-3218-46fb-bfd7-b193dccc8109-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.083450 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.083317    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9b2053-3218-46fb-bfd7-b193dccc8109-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.083450 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.083390    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9b2053-3218-46fb-bfd7-b193dccc8109-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.083701 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.083452    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sprw5\" (UniqueName: \"kubernetes.io/projected/4c9b2053-3218-46fb-bfd7-b193dccc8109-kube-api-access-sprw5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.083701 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.083503    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wnx4\" (UniqueName: \"kubernetes.io/projected/096035a0-30e5-4124-ad5b-8bfe177aebf1-kube-api-access-2wnx4\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.083701 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.083531    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/096035a0-30e5-4124-ad5b-8bfe177aebf1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.083701 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.083559    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/096035a0-30e5-4124-ad5b-8bfe177aebf1-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.083701 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.083617    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9b2053-3218-46fb-bfd7-b193dccc8109-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.083701 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.083654    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/096035a0-30e5-4124-ad5b-8bfe177aebf1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.083701 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.083695    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/096035a0-30e5-4124-ad5b-8bfe177aebf1-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.084158 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.083794    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/096035a0-30e5-4124-ad5b-8bfe177aebf1-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.184871 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.184776    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wnx4\" (UniqueName: \"kubernetes.io/projected/096035a0-30e5-4124-ad5b-8bfe177aebf1-kube-api-access-2wnx4\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.184871 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.184824    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/096035a0-30e5-4124-ad5b-8bfe177aebf1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.184871 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.184857    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/096035a0-30e5-4124-ad5b-8bfe177aebf1-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.185305 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.184888    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9b2053-3218-46fb-bfd7-b193dccc8109-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.185305 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.184918    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/096035a0-30e5-4124-ad5b-8bfe177aebf1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.185305 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.184958    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/096035a0-30e5-4124-ad5b-8bfe177aebf1-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.185305 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.185055    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/096035a0-30e5-4124-ad5b-8bfe177aebf1-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.185305 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.185096    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9b2053-3218-46fb-bfd7-b193dccc8109-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.185305 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.185126    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/096035a0-30e5-4124-ad5b-8bfe177aebf1-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.185305 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.185152    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/096035a0-30e5-4124-ad5b-8bfe177aebf1-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.185305 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.185187    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9b2053-3218-46fb-bfd7-b193dccc8109-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.185305 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.185221    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9b2053-3218-46fb-bfd7-b193dccc8109-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.185305 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.185249    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/096035a0-30e5-4124-ad5b-8bfe177aebf1-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.185305 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.185291    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9b2053-3218-46fb-bfd7-b193dccc8109-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.186140 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.185320    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9b2053-3218-46fb-bfd7-b193dccc8109-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.186140 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.185381    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9b2053-3218-46fb-bfd7-b193dccc8109-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.186140 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.185415    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9b2053-3218-46fb-bfd7-b193dccc8109-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.186140 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.185443    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sprw5\" (UniqueName: \"kubernetes.io/projected/4c9b2053-3218-46fb-bfd7-b193dccc8109-kube-api-access-sprw5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.186140 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.185583    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/096035a0-30e5-4124-ad5b-8bfe177aebf1-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.186140 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.185685    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/096035a0-30e5-4124-ad5b-8bfe177aebf1-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.186140 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.185693    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/096035a0-30e5-4124-ad5b-8bfe177aebf1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.186140 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.185899    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9b2053-3218-46fb-bfd7-b193dccc8109-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.186140 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.185968    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c9b2053-3218-46fb-bfd7-b193dccc8109-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.186140 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.185970    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/096035a0-30e5-4124-ad5b-8bfe177aebf1-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.186746 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.186152    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c9b2053-3218-46fb-bfd7-b193dccc8109-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.186746 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.186383    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9b2053-3218-46fb-bfd7-b193dccc8109-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.187061 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.187012    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c9b2053-3218-46fb-bfd7-b193dccc8109-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.187569 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.187543    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/096035a0-30e5-4124-ad5b-8bfe177aebf1-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.188498 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.188472    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/096035a0-30e5-4124-ad5b-8bfe177aebf1-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.188769 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.188751    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/096035a0-30e5-4124-ad5b-8bfe177aebf1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.189046 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.189018    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c9b2053-3218-46fb-bfd7-b193dccc8109-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.189383 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.189363    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c9b2053-3218-46fb-bfd7-b193dccc8109-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.240832 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.240806    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9b2053-3218-46fb-bfd7-b193dccc8109-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.241061 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.241039    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sprw5\" (UniqueName: \"kubernetes.io/projected/4c9b2053-3218-46fb-bfd7-b193dccc8109-kube-api-access-sprw5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-7s45f\" (UID: \"4c9b2053-3218-46fb-bfd7-b193dccc8109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.241170 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.241150    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/096035a0-30e5-4124-ad5b-8bfe177aebf1-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.241756 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.241740    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wnx4\" (UniqueName: \"kubernetes.io/projected/096035a0-30e5-4124-ad5b-8bfe177aebf1-kube-api-access-2wnx4\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-r6p52\" (UID: \"096035a0-30e5-4124-ad5b-8bfe177aebf1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.251601 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.251584    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:19.509291 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.509259    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:19.668869 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.668836    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"]
Nov 03 00:23:19.671262 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:23:19.671236    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9b2053_3218_46fb_bfd7_b193dccc8109.slice/crio-b9ed25107cacfcd9af75617a3961fd3ca22e751347925d50bf2ccd2bc46f6aff WatchSource:0}: Error finding container b9ed25107cacfcd9af75617a3961fd3ca22e751347925d50bf2ccd2bc46f6aff: Status 404 returned error can't find the container with id b9ed25107cacfcd9af75617a3961fd3ca22e751347925d50bf2ccd2bc46f6aff
Nov 03 00:23:19.674192 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.674160    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:23:19.674303 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.674221    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:23:19.674303 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.674249    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:23:19.881284 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.881254    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"]
Nov 03 00:23:19.882588 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:23:19.882548    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096035a0_30e5_4124_ad5b_8bfe177aebf1.slice/crio-ea6815a880a80b69cc3aebe0a8f3416cec01a7d141307ee6dc0641d601549575 WatchSource:0}: Error finding container ea6815a880a80b69cc3aebe0a8f3416cec01a7d141307ee6dc0641d601549575: Status 404 returned error can't find the container with id ea6815a880a80b69cc3aebe0a8f3416cec01a7d141307ee6dc0641d601549575
Nov 03 00:23:19.884949 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.884911    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:23:19.885049 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.885012    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:23:19.885104 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:19.885063    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:23:20.328725 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:20.328691    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52" event={"ID":"096035a0-30e5-4124-ad5b-8bfe177aebf1","Type":"ContainerStarted","Data":"d756b46f453bda74c33054b9a3d5295f696a43282a1d6580a593fa1d15189e60"}
Nov 03 00:23:20.329258 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:20.328750    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52" event={"ID":"096035a0-30e5-4124-ad5b-8bfe177aebf1","Type":"ContainerStarted","Data":"ea6815a880a80b69cc3aebe0a8f3416cec01a7d141307ee6dc0641d601549575"}
Nov 03 00:23:20.330411 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:20.330383    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f" event={"ID":"4c9b2053-3218-46fb-bfd7-b193dccc8109","Type":"ContainerStarted","Data":"6cface6ad91a3d6c63a73e58f0547a9edaead87ef867a546edee54a9be0bca36"}
Nov 03 00:23:20.330557 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:20.330436    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f" event={"ID":"4c9b2053-3218-46fb-bfd7-b193dccc8109","Type":"ContainerStarted","Data":"b9ed25107cacfcd9af75617a3961fd3ca22e751347925d50bf2ccd2bc46f6aff"}
Nov 03 00:23:20.446140 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:20.446074    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52" podStartSLOduration=2.446055603 podStartE2EDuration="2.446055603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:20.44434724 +0000 UTC m=+5440.012702026" watchObservedRunningTime="2026-11-03 00:23:20.446055603 +0000 UTC m=+5440.014410391"
Nov 03 00:23:20.509820 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:20.509786    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:20.511080 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:20.511057    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-r6p52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 00:23:20.511244 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:20.511224    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52" podUID="096035a0-30e5-4124-ad5b-8bfe177aebf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 00:23:20.576004 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:20.566190    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f" podStartSLOduration=2.56617009 podStartE2EDuration="2.56617009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:20.565887645 +0000 UTC m=+5440.134242432" watchObservedRunningTime="2026-11-03 00:23:20.56617009 +0000 UTC m=+5440.134524879"
Nov 03 00:23:21.091017 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:21.087947    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:21.246603 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:21.246556    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:21.254019 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:21.252593    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:21.267000 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:21.264125    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7s45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Nov 03 00:23:21.267000 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:21.264217    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f" podUID="4c9b2053-3218-46fb-bfd7-b193dccc8109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Nov 03 00:23:21.510494 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:21.510458    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-r6p52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 00:23:21.510896 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:21.510521    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52" podUID="096035a0-30e5-4124-ad5b-8bfe177aebf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 00:23:22.252360 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:22.252317    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7s45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Nov 03 00:23:22.252563 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:22.252377    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f" podUID="4c9b2053-3218-46fb-bfd7-b193dccc8109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Nov 03 00:23:22.510629 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:22.510537    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-r6p52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 00:23:22.510629 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:22.510612    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52" podUID="096035a0-30e5-4124-ad5b-8bfe177aebf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 00:23:23.252396 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:23.252358    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7s45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Nov 03 00:23:23.252596 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:23.252439    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f" podUID="4c9b2053-3218-46fb-bfd7-b193dccc8109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Nov 03 00:23:23.510051 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:23.509946    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-r6p52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 00:23:23.510051 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:23.510031    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52" podUID="096035a0-30e5-4124-ad5b-8bfe177aebf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 00:23:24.252491 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:24.252454    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7s45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Nov 03 00:23:24.253010 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:24.252512    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f" podUID="4c9b2053-3218-46fb-bfd7-b193dccc8109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Nov 03 00:23:24.510525 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:24.510435    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-r6p52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 00:23:24.510525 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:24.510487    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52" podUID="096035a0-30e5-4124-ad5b-8bfe177aebf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 00:23:25.252732 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:25.252696    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7s45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Nov 03 00:23:25.253194 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:25.252767    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f" podUID="4c9b2053-3218-46fb-bfd7-b193dccc8109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Nov 03 00:23:25.510189 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:25.510101    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-r6p52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 00:23:25.510189 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:25.510151    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52" podUID="096035a0-30e5-4124-ad5b-8bfe177aebf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 00:23:26.252667 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:26.252632    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7s45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Nov 03 00:23:26.252824 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:26.252684    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f" podUID="4c9b2053-3218-46fb-bfd7-b193dccc8109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Nov 03 00:23:26.509863 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:26.509780    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-r6p52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 00:23:26.509863 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:26.509843    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52" podUID="096035a0-30e5-4124-ad5b-8bfe177aebf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 00:23:27.252582 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:27.252546    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7s45f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Nov 03 00:23:27.252776 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:27.252604    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f" podUID="4c9b2053-3218-46fb-bfd7-b193dccc8109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Nov 03 00:23:27.510480 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:27.510391    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-r6p52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 00:23:27.511023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:27.510502    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52" podUID="096035a0-30e5-4124-ad5b-8bfe177aebf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 00:23:28.257052 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:28.257020    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:28.257313 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:28.257285    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:28.258154 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:28.258133    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f"
Nov 03 00:23:28.513737 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:28.513661    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:28.514232 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:28.514014    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:28.514616 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:28.514599    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52"
Nov 03 00:23:41.749954 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:23:41.749913    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:23:52.484324 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:52.484288    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:53.290751 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:23:53.290717    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:07.241308 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:07.241269    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:07.399299 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:07.399261    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:11.756200 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:24:11.756084    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:24:13.739119 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:13.739085    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"]
Nov 03 00:24:13.742918 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:13.742896    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:13.832006 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:13.831480    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"]
Nov 03 00:24:13.840503 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:13.840475    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-78f9ed86-kserve-self-signed-certs\""
Nov 03 00:24:13.863417 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:13.863390    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:13.932348 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:13.932313    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-model-cache\") pod \"e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:13.932565 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:13.932356    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4v9s\" (UniqueName: \"kubernetes.io/projected/296d0834-fbe7-45cc-a109-4d5297d07f78-kube-api-access-m4v9s\") pod \"e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:13.932565 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:13.932378    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-home\") pod \"e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:13.932731 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:13.932649    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-tmp-dir\") pod \"e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:13.932832 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:13.932799    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-dshm\") pod \"e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:13.932909 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:13.932870    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/296d0834-fbe7-45cc-a109-4d5297d07f78-tls-certs\") pod \"e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:14.034099 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.034070    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-tmp-dir\") pod \"e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:14.034259 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.034131    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-dshm\") pod \"e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:14.034259 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.034157    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/296d0834-fbe7-45cc-a109-4d5297d07f78-tls-certs\") pod \"e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:14.034259 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.034174    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-model-cache\") pod \"e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:14.034259 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.034195    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4v9s\" (UniqueName: \"kubernetes.io/projected/296d0834-fbe7-45cc-a109-4d5297d07f78-kube-api-access-m4v9s\") pod \"e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:14.034259 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.034213    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-home\") pod \"e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:14.034564 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.034541    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-tmp-dir\") pod \"e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:14.034633 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.034613    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-model-cache\") pod \"e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:14.034688 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.034631    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-home\") pod \"e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:14.036524 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.036495    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-dshm\") pod \"e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:14.036774 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.036755    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/296d0834-fbe7-45cc-a109-4d5297d07f78-tls-certs\") pod \"e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:14.088360 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.088334    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4v9s\" (UniqueName: \"kubernetes.io/projected/296d0834-fbe7-45cc-a109-4d5297d07f78-kube-api-access-m4v9s\") pod \"e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:14.332791 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.332720    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j"]
Nov 03 00:24:14.336927 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.336912    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j"
Nov 03 00:24:14.355105 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.355084    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:14.387823 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.387791    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-78f9ed86-epp-sa-dockercfg-srbtn\""
Nov 03 00:24:14.404219 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.404187    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j"]
Nov 03 00:24:14.437433 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.437403    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj2r6\" (UniqueName: \"kubernetes.io/projected/32789441-6263-4e6e-a2ce-efe240130a6f-kube-api-access-bj2r6\") pod \"e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j\" (UID: \"32789441-6263-4e6e-a2ce-efe240130a6f\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j"
Nov 03 00:24:14.437584 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.437518    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32789441-6263-4e6e-a2ce-efe240130a6f-tls-certs\") pod \"e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j\" (UID: \"32789441-6263-4e6e-a2ce-efe240130a6f\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j"
Nov 03 00:24:14.492525 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.492482    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:14.538932 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.538903    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32789441-6263-4e6e-a2ce-efe240130a6f-tls-certs\") pod \"e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j\" (UID: \"32789441-6263-4e6e-a2ce-efe240130a6f\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j"
Nov 03 00:24:14.539124 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.538955    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj2r6\" (UniqueName: \"kubernetes.io/projected/32789441-6263-4e6e-a2ce-efe240130a6f-kube-api-access-bj2r6\") pod \"e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j\" (UID: \"32789441-6263-4e6e-a2ce-efe240130a6f\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j"
Nov 03 00:24:14.541446 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.541421    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32789441-6263-4e6e-a2ce-efe240130a6f-tls-certs\") pod \"e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j\" (UID: \"32789441-6263-4e6e-a2ce-efe240130a6f\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j"
Nov 03 00:24:14.596355 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.596297    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj2r6\" (UniqueName: \"kubernetes.io/projected/32789441-6263-4e6e-a2ce-efe240130a6f-kube-api-access-bj2r6\") pod \"e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j\" (UID: \"32789441-6263-4e6e-a2ce-efe240130a6f\") " pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j"
Nov 03 00:24:14.650752 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.650728    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j"
Nov 03 00:24:14.729679 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:14.729657    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"]
Nov 03 00:24:14.730634 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:24:14.730609    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod296d0834_fbe7_45cc_a109_4d5297d07f78.slice/crio-e9f532a657b0970d85f251491e4bc3698278cbe3deea2abd1f58c3ab66a97edf WatchSource:0}: Error finding container e9f532a657b0970d85f251491e4bc3698278cbe3deea2abd1f58c3ab66a97edf: Status 404 returned error can't find the container with id e9f532a657b0970d85f251491e4bc3698278cbe3deea2abd1f58c3ab66a97edf
Nov 03 00:24:15.015650 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:15.015623    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j"]
Nov 03 00:24:15.016688 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:24:15.016655    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32789441_6263_4e6e_a2ce_efe240130a6f.slice/crio-38d413cbc2f6982fee8c3dfe44ea4bbdd84b4e70536b0236e8fd8eaa249b2308 WatchSource:0}: Error finding container 38d413cbc2f6982fee8c3dfe44ea4bbdd84b4e70536b0236e8fd8eaa249b2308: Status 404 returned error can't find the container with id 38d413cbc2f6982fee8c3dfe44ea4bbdd84b4e70536b0236e8fd8eaa249b2308
Nov 03 00:24:15.018826 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:15.018811    4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:24:15.641746 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:15.641715    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j" event={"ID":"32789441-6263-4e6e-a2ce-efe240130a6f","Type":"ContainerStarted","Data":"38d413cbc2f6982fee8c3dfe44ea4bbdd84b4e70536b0236e8fd8eaa249b2308"}
Nov 03 00:24:15.643309 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:15.643284    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68" event={"ID":"296d0834-fbe7-45cc-a109-4d5297d07f78","Type":"ContainerStarted","Data":"2742471e60e2e5719d4b0c8e6ac240f17e6eb763ca14c6d0dbc21e1d15f24050"}
Nov 03 00:24:15.643449 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:15.643317    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68" event={"ID":"296d0834-fbe7-45cc-a109-4d5297d07f78","Type":"ContainerStarted","Data":"e9f532a657b0970d85f251491e4bc3698278cbe3deea2abd1f58c3ab66a97edf"}
Nov 03 00:24:15.754554 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:15.754502    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68" podStartSLOduration=2.754490464 podStartE2EDuration="2.754490464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:15.752875192 +0000 UTC m=+5495.321229990" watchObservedRunningTime="2026-11-03 00:24:15.754490464 +0000 UTC m=+5495.322845251"
Nov 03 00:24:16.355563 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:16.355530    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:17.654294 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:17.654200    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j" event={"ID":"32789441-6263-4e6e-a2ce-efe240130a6f","Type":"ContainerStarted","Data":"7186fd38fe36f025758e7a2d0b290b5407cf9b0ea25208e5bb58aba9375cd745"}
Nov 03 00:24:17.654760 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:17.654290    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j"
Nov 03 00:24:17.765519 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:17.765463    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j" podStartSLOduration=1.528426844 podStartE2EDuration="3.765451472s" totalImagesPullingTime="2.237024628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:17.765103706 +0000 UTC m=+5497.333458493" watchObservedRunningTime="2026-11-03 00:24:17.765451472 +0000 UTC m=+5497.333806259"
Nov 03 00:24:18.925492 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:18.925453    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:20.885004 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:20.882866    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:24.356081 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:24.356043    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:24.375035 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:24.375004    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:24.727211 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:24.727127    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:24:41.749337 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:24:41.749301    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:24:48.660379 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:48.660346    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j"
Nov 03 00:24:52.785749 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:24:52.785710    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:11.749756 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:25:11.749664    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:25:21.895299 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:21.895264    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"]
Nov 03 00:25:21.904967 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:21.904937    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j"]
Nov 03 00:25:21.939686 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:21.939654    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:22.103549 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:22.103516    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:28.688006 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:28.685135    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j" podUID="32789441-6263-4e6e-a2ce-efe240130a6f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:25:29.332025 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:29.316162    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-7s45f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:25:29.332025 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:29.316232    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-7s45f" podUID="4c9b2053-3218-46fb-bfd7-b193dccc8109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:25:29.570810 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:29.570766    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-r6p52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:25:29.571130 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:29.571097    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52" podUID="096035a0-30e5-4124-ad5b-8bfe177aebf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:25:36.921053 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:36.920971    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68" podUID="296d0834-fbe7-45cc-a109-4d5297d07f78" containerName="main" containerID="cri-o://2742471e60e2e5719d4b0c8e6ac240f17e6eb763ca14c6d0dbc21e1d15f24050" gracePeriod=45
Nov 03 00:25:36.924436 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:36.924389    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j" podUID="32789441-6263-4e6e-a2ce-efe240130a6f" containerName="main" containerID="cri-o://7186fd38fe36f025758e7a2d0b290b5407cf9b0ea25208e5bb58aba9375cd745" gracePeriod=45
Nov 03 00:25:37.094361 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.094328    4294 generic.go:356] "Generic (PLEG): container finished" podID="296d0834-fbe7-45cc-a109-4d5297d07f78" containerID="2742471e60e2e5719d4b0c8e6ac240f17e6eb763ca14c6d0dbc21e1d15f24050" exitCode=0
Nov 03 00:25:37.094506 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.094396    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68" event={"ID":"296d0834-fbe7-45cc-a109-4d5297d07f78","Type":"ContainerDied","Data":"2742471e60e2e5719d4b0c8e6ac240f17e6eb763ca14c6d0dbc21e1d15f24050"}
Nov 03 00:25:37.172617 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.172549    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:25:37.210562 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.210533    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-dshm\" (UniqueName: \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-dshm\") pod \"296d0834-fbe7-45cc-a109-4d5297d07f78\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") "
Nov 03 00:25:37.210681 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.210620    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/296d0834-fbe7-45cc-a109-4d5297d07f78-tls-certs\" (UniqueName: \"kubernetes.io/secret/296d0834-fbe7-45cc-a109-4d5297d07f78-tls-certs\") pod \"296d0834-fbe7-45cc-a109-4d5297d07f78\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") "
Nov 03 00:25:37.210681 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.210655    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-tmp-dir\") pod \"296d0834-fbe7-45cc-a109-4d5297d07f78\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") "
Nov 03 00:25:37.210788 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.210692    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-model-cache\") pod \"296d0834-fbe7-45cc-a109-4d5297d07f78\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") "
Nov 03 00:25:37.210788 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.210728    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/296d0834-fbe7-45cc-a109-4d5297d07f78-kube-api-access-m4v9s\" (UniqueName: \"kubernetes.io/projected/296d0834-fbe7-45cc-a109-4d5297d07f78-kube-api-access-m4v9s\") pod \"296d0834-fbe7-45cc-a109-4d5297d07f78\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") "
Nov 03 00:25:37.210788 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.210754    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-home\" (UniqueName: \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-home\") pod \"296d0834-fbe7-45cc-a109-4d5297d07f78\" (UID: \"296d0834-fbe7-45cc-a109-4d5297d07f78\") "
Nov 03 00:25:37.211031 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.210933    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-tmp-dir" pod "296d0834-fbe7-45cc-a109-4d5297d07f78" (UID: "296d0834-fbe7-45cc-a109-4d5297d07f78"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:25:37.211031 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.211016    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-model-cache" pod "296d0834-fbe7-45cc-a109-4d5297d07f78" (UID: "296d0834-fbe7-45cc-a109-4d5297d07f78"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:25:37.211223 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.211090    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-home" pod "296d0834-fbe7-45cc-a109-4d5297d07f78" (UID: "296d0834-fbe7-45cc-a109-4d5297d07f78"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:25:37.211223 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.211111    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:25:37.211223 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.211133    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:25:37.212961 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.212941    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296d0834-fbe7-45cc-a109-4d5297d07f78-kube-api-access-m4v9s" pod "296d0834-fbe7-45cc-a109-4d5297d07f78" (UID: "296d0834-fbe7-45cc-a109-4d5297d07f78"). InnerVolumeSpecName "kube-api-access-m4v9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:25:37.213191 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.213173    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/296d0834-fbe7-45cc-a109-4d5297d07f78-tls-certs" pod "296d0834-fbe7-45cc-a109-4d5297d07f78" (UID: "296d0834-fbe7-45cc-a109-4d5297d07f78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:25:37.213264 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.213178    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-dshm" pod "296d0834-fbe7-45cc-a109-4d5297d07f78" (UID: "296d0834-fbe7-45cc-a109-4d5297d07f78"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:25:37.312049 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.312014    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:25:37.312049 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.312044    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/296d0834-fbe7-45cc-a109-4d5297d07f78-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:25:37.312049 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.312056    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4v9s\" (UniqueName: \"kubernetes.io/projected/296d0834-fbe7-45cc-a109-4d5297d07f78-kube-api-access-m4v9s\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:25:37.312286 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:37.312069    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/296d0834-fbe7-45cc-a109-4d5297d07f78-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:25:38.099810 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:38.099779    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"
Nov 03 00:25:38.099810 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:38.099797    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68" event={"ID":"296d0834-fbe7-45cc-a109-4d5297d07f78","Type":"ContainerDied","Data":"e9f532a657b0970d85f251491e4bc3698278cbe3deea2abd1f58c3ab66a97edf"}
Nov 03 00:25:38.100334 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:38.099853    4294 scope.go:122] "RemoveContainer" containerID="2742471e60e2e5719d4b0c8e6ac240f17e6eb763ca14c6d0dbc21e1d15f24050"
Nov 03 00:25:38.258961 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:38.258929    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"]
Nov 03 00:25:38.306653 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:38.306622    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-78f9ed86-kserve-bdd5f969d-rzm68"]
Nov 03 00:25:38.546244 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:38.546214    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="296d0834-fbe7-45cc-a109-4d5297d07f78" path="/var/lib/kubelet/pods/296d0834-fbe7-45cc-a109-4d5297d07f78/volumes"
Nov 03 00:25:38.661132 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:38.661096    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j" podUID="32789441-6263-4e6e-a2ce-efe240130a6f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:25:40.078093 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:40.078062    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:40.683817 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:40.683770    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:40.911416 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:40.911383    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:41.748674 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:25:41.748634    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:25:44.566109 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:44.566073    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-r6p52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:25:44.566695 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:44.566650    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-r6p52" podUID="096035a0-30e5-4124-ad5b-8bfe177aebf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:25:44.931813 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:44.931532    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:45.131518 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:45.131474    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:45.316021 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:45.315990    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:48.143109 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:48.143073    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:48.318224 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:48.318190    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:48.661029 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:48.660973    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j" podUID="32789441-6263-4e6e-a2ce-efe240130a6f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:25:48.661210 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:48.661095    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j"
Nov 03 00:25:51.927175 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:51.927133    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:52.116699 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:52.116658    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:52.291446 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:52.291394    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:53.125295 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:53.125254    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:57.088405 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:57.088372    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:58.660555 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:25:58.660517    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j" podUID="32789441-6263-4e6e-a2ce-efe240130a6f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:26:02.295995 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:02.295951    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:07.170423 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:07.170397    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j"
Nov 03 00:26:07.231734 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:07.231652    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32789441-6263-4e6e-a2ce-efe240130a6f-kube-api-access-bj2r6\" (UniqueName: \"kubernetes.io/projected/32789441-6263-4e6e-a2ce-efe240130a6f-kube-api-access-bj2r6\") pod \"32789441-6263-4e6e-a2ce-efe240130a6f\" (UID: \"32789441-6263-4e6e-a2ce-efe240130a6f\") "
Nov 03 00:26:07.231734 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:07.231694    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32789441-6263-4e6e-a2ce-efe240130a6f-tls-certs\" (UniqueName: \"kubernetes.io/secret/32789441-6263-4e6e-a2ce-efe240130a6f-tls-certs\") pod \"32789441-6263-4e6e-a2ce-efe240130a6f\" (UID: \"32789441-6263-4e6e-a2ce-efe240130a6f\") "
Nov 03 00:26:07.233883 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:07.233850    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32789441-6263-4e6e-a2ce-efe240130a6f-tls-certs" pod "32789441-6263-4e6e-a2ce-efe240130a6f" (UID: "32789441-6263-4e6e-a2ce-efe240130a6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:26:07.234036 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:07.233892    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32789441-6263-4e6e-a2ce-efe240130a6f-kube-api-access-bj2r6" pod "32789441-6263-4e6e-a2ce-efe240130a6f" (UID: "32789441-6263-4e6e-a2ce-efe240130a6f"). InnerVolumeSpecName "kube-api-access-bj2r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:26:07.237294 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:07.237270    4294 generic.go:356] "Generic (PLEG): container finished" podID="32789441-6263-4e6e-a2ce-efe240130a6f" containerID="7186fd38fe36f025758e7a2d0b290b5407cf9b0ea25208e5bb58aba9375cd745" exitCode=0
Nov 03 00:26:07.237405 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:07.237312    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j"
Nov 03 00:26:07.237405 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:07.237353    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j" event={"ID":"32789441-6263-4e6e-a2ce-efe240130a6f","Type":"ContainerDied","Data":"7186fd38fe36f025758e7a2d0b290b5407cf9b0ea25208e5bb58aba9375cd745"}
Nov 03 00:26:07.237405 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:07.237403    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j" event={"ID":"32789441-6263-4e6e-a2ce-efe240130a6f","Type":"ContainerDied","Data":"38d413cbc2f6982fee8c3dfe44ea4bbdd84b4e70536b0236e8fd8eaa249b2308"}
Nov 03 00:26:07.237523 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:07.237427    4294 scope.go:122] "RemoveContainer" containerID="7186fd38fe36f025758e7a2d0b290b5407cf9b0ea25208e5bb58aba9375cd745"
Nov 03 00:26:07.252745 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:07.252728    4294 scope.go:122] "RemoveContainer" containerID="7186fd38fe36f025758e7a2d0b290b5407cf9b0ea25208e5bb58aba9375cd745"
Nov 03 00:26:07.253022 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:26:07.253005    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7186fd38fe36f025758e7a2d0b290b5407cf9b0ea25208e5bb58aba9375cd745\": container with ID starting with 7186fd38fe36f025758e7a2d0b290b5407cf9b0ea25208e5bb58aba9375cd745 not found: ID does not exist" containerID="7186fd38fe36f025758e7a2d0b290b5407cf9b0ea25208e5bb58aba9375cd745"
Nov 03 00:26:07.253094 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:07.253033    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7186fd38fe36f025758e7a2d0b290b5407cf9b0ea25208e5bb58aba9375cd745"} err="failed to get container status \"7186fd38fe36f025758e7a2d0b290b5407cf9b0ea25208e5bb58aba9375cd745\": rpc error: code = NotFound desc = could not find container \"7186fd38fe36f025758e7a2d0b290b5407cf9b0ea25208e5bb58aba9375cd745\": container with ID starting with 7186fd38fe36f025758e7a2d0b290b5407cf9b0ea25208e5bb58aba9375cd745 not found: ID does not exist"
Nov 03 00:26:07.332344 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:07.332312    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj2r6\" (UniqueName: \"kubernetes.io/projected/32789441-6263-4e6e-a2ce-efe240130a6f-kube-api-access-bj2r6\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:26:07.332344 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:07.332341    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32789441-6263-4e6e-a2ce-efe240130a6f-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:26:07.388158 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:07.388125    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j"]
Nov 03 00:26:07.392118 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:07.392095    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-78f9ed86-kserve-router-scheduler-5d8ccd5c85-9sf5j"]
Nov 03 00:26:07.411663 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:07.411614    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:08.546599 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:08.546565    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32789441-6263-4e6e-a2ce-efe240130a6f" path="/var/lib/kubelet/pods/32789441-6263-4e6e-a2ce-efe240130a6f/volumes"
Nov 03 00:26:09.565135 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:09.565086    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:10.162563 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:10.162532    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:10.333903 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:10.333868    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:11.774669 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:26:11.774623    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:26:12.856053 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:12.856015    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:13.310102 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.310070    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"]
Nov 03 00:26:13.310455 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.310443    4294 cpu_manager.go:414] "containerMap: removing container" podUID="296d0834-fbe7-45cc-a109-4d5297d07f78" containerName="main"
Nov 03 00:26:13.310507 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.310456    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="296d0834-fbe7-45cc-a109-4d5297d07f78" containerName="main"
Nov 03 00:26:13.310507 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.310485    4294 cpu_manager.go:414] "containerMap: removing container" podUID="32789441-6263-4e6e-a2ce-efe240130a6f" containerName="main"
Nov 03 00:26:13.310507 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.310490    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32789441-6263-4e6e-a2ce-efe240130a6f" containerName="main"
Nov 03 00:26:13.310630 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.310552    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="296d0834-fbe7-45cc-a109-4d5297d07f78" containerName="main"
Nov 03 00:26:13.310630 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.310567    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="32789441-6263-4e6e-a2ce-efe240130a6f" containerName="main"
Nov 03 00:26:13.314962 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.314947    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:13.366971 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.366939    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"]
Nov 03 00:26:13.390135 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.390101    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40799d7b-f17b-4a61-b95b-13330eda3101-tls-certs\") pod \"simulator-single-main-adc00e-kserve-8dbdd7875-nl25w\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") " pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:13.390135 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.390138    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pnbs\" (UniqueName: \"kubernetes.io/projected/40799d7b-f17b-4a61-b95b-13330eda3101-kube-api-access-9pnbs\") pod \"simulator-single-main-adc00e-kserve-8dbdd7875-nl25w\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") " pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:13.390344 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.390160    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-dshm\") pod \"simulator-single-main-adc00e-kserve-8dbdd7875-nl25w\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") " pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:13.390344 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.390272    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-home\") pod \"simulator-single-main-adc00e-kserve-8dbdd7875-nl25w\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") " pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:13.390344 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.390303    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-tmp-dir\") pod \"simulator-single-main-adc00e-kserve-8dbdd7875-nl25w\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") " pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:13.390455 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.390427    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-model-cache\") pod \"simulator-single-main-adc00e-kserve-8dbdd7875-nl25w\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") " pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:13.412742 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.412714    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-adc00e\"/\"openshift-service-ca.crt\""
Nov 03 00:26:13.412883 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.412752    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-adc00e\"/\"kube-root-ca.crt\""
Nov 03 00:26:13.412883 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.412796    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-adc00e\"/\"default-dockercfg-hrp2l\""
Nov 03 00:26:13.413281 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.413265    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-adc00e\"/\"simulator-single-main-adc00e-kserve-self-signed-certs\""
Nov 03 00:26:13.424563 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.424537    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:13.491486 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.491444    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-dshm\") pod \"simulator-single-main-adc00e-kserve-8dbdd7875-nl25w\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") " pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:13.491708 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.491500    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-home\") pod \"simulator-single-main-adc00e-kserve-8dbdd7875-nl25w\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") " pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:13.491708 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.491539    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-tmp-dir\") pod \"simulator-single-main-adc00e-kserve-8dbdd7875-nl25w\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") " pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:13.491708 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.491639    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-model-cache\") pod \"simulator-single-main-adc00e-kserve-8dbdd7875-nl25w\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") " pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:13.491708 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.491701    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40799d7b-f17b-4a61-b95b-13330eda3101-tls-certs\") pod \"simulator-single-main-adc00e-kserve-8dbdd7875-nl25w\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") " pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:13.491954 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.491732    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pnbs\" (UniqueName: \"kubernetes.io/projected/40799d7b-f17b-4a61-b95b-13330eda3101-kube-api-access-9pnbs\") pod \"simulator-single-main-adc00e-kserve-8dbdd7875-nl25w\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") " pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:13.492041 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.492008    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-home\") pod \"simulator-single-main-adc00e-kserve-8dbdd7875-nl25w\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") " pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:13.492308 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.492275    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-tmp-dir\") pod \"simulator-single-main-adc00e-kserve-8dbdd7875-nl25w\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") " pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:13.492449 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.492398    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-model-cache\") pod \"simulator-single-main-adc00e-kserve-8dbdd7875-nl25w\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") " pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:13.493842 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.493815    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-dshm\") pod \"simulator-single-main-adc00e-kserve-8dbdd7875-nl25w\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") " pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:13.494114 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.494094    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40799d7b-f17b-4a61-b95b-13330eda3101-tls-certs\") pod \"simulator-single-main-adc00e-kserve-8dbdd7875-nl25w\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") " pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:13.547056 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.547033    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pnbs\" (UniqueName: \"kubernetes.io/projected/40799d7b-f17b-4a61-b95b-13330eda3101-kube-api-access-9pnbs\") pod \"simulator-single-main-adc00e-kserve-8dbdd7875-nl25w\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") " pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:13.626879 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.626797    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:13.993218 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:13.993192    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"]
Nov 03 00:26:13.995147 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:26:13.995108    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40799d7b_f17b_4a61_b95b_13330eda3101.slice/crio-e6504f68f6190666ec13d42084a1ed88cfae4e35eeb1594b1adcbc4c61707bc7 WatchSource:0}: Error finding container e6504f68f6190666ec13d42084a1ed88cfae4e35eeb1594b1adcbc4c61707bc7: Status 404 returned error can't find the container with id e6504f68f6190666ec13d42084a1ed88cfae4e35eeb1594b1adcbc4c61707bc7
Nov 03 00:26:14.274087 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:14.274011    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w" event={"ID":"40799d7b-f17b-4a61-b95b-13330eda3101","Type":"ContainerStarted","Data":"e378232d96dae09f9eeb6947f62aa81f2c1292da9a0475922d73bdd30dcc3b58"}
Nov 03 00:26:14.274087 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:14.274057    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w" event={"ID":"40799d7b-f17b-4a61-b95b-13330eda3101","Type":"ContainerStarted","Data":"e6504f68f6190666ec13d42084a1ed88cfae4e35eeb1594b1adcbc4c61707bc7"}
Nov 03 00:26:14.385861 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:14.385810    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w" podStartSLOduration=1.3857973000000001 podStartE2EDuration="1.3857973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:14.384131259 +0000 UTC m=+5613.952486047" watchObservedRunningTime="2026-11-03 00:26:14.3857973 +0000 UTC m=+5613.954152086"
Nov 03 00:26:14.627532 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:14.627457    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:17.942989 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:17.942927    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:22.969657 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:22.969622    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:23.627399 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:23.627368    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:23.639907 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:23.639848    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:24.331741 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:24.331712    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:26:28.011070 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:28.011035    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:32.981027 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:32.980967    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:37.960043 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:37.960003    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:41.749745 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:26:41.749708    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:26:42.981521 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:42.981478    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:47.978592 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:47.978555    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:52.968723 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:52.968684    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:53.486267 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:53.486227    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:57.968680 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:26:57.968644    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:02.994274 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:02.994239    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:07.896906 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:07.896871    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:09.812279 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:09.812237    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:10.266918 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:10.266887    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:11.749314 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:27:11.749273    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:27:12.892848 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:12.892817    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:14.734254 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:14.734223    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"]
Nov 03 00:27:14.838739 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:14.838707    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:16.156548 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:16.156503    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:17.013668 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:17.013632    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:17.187323 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:17.187286    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:18.116448 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:18.116416    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:21.862281 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:21.862244    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:22.696483 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:22.696439    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:27.890190 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:27.890158    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:28.063215 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:28.063177    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:29.754449 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:29.754406    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w" podUID="40799d7b-f17b-4a61-b95b-13330eda3101" containerName="main" containerID="cri-o://e378232d96dae09f9eeb6947f62aa81f2c1292da9a0475922d73bdd30dcc3b58" gracePeriod=45
Nov 03 00:27:30.000521 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.000495    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:27:30.016680 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.016602    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-model-cache\") pod \"40799d7b-f17b-4a61-b95b-13330eda3101\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") "
Nov 03 00:27:30.016831 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.016696    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-tmp-dir\") pod \"40799d7b-f17b-4a61-b95b-13330eda3101\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") "
Nov 03 00:27:30.016831 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.016749    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-home\" (UniqueName: \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-home\") pod \"40799d7b-f17b-4a61-b95b-13330eda3101\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") "
Nov 03 00:27:30.016831 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.016795    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40799d7b-f17b-4a61-b95b-13330eda3101-tls-certs\" (UniqueName: \"kubernetes.io/secret/40799d7b-f17b-4a61-b95b-13330eda3101-tls-certs\") pod \"40799d7b-f17b-4a61-b95b-13330eda3101\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") "
Nov 03 00:27:30.016955 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.016881    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-dshm\" (UniqueName: \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-dshm\") pod \"40799d7b-f17b-4a61-b95b-13330eda3101\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") "
Nov 03 00:27:30.016955 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.016910    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-model-cache" pod "40799d7b-f17b-4a61-b95b-13330eda3101" (UID: "40799d7b-f17b-4a61-b95b-13330eda3101"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:30.017055 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.016972    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40799d7b-f17b-4a61-b95b-13330eda3101-kube-api-access-9pnbs\" (UniqueName: \"kubernetes.io/projected/40799d7b-f17b-4a61-b95b-13330eda3101-kube-api-access-9pnbs\") pod \"40799d7b-f17b-4a61-b95b-13330eda3101\" (UID: \"40799d7b-f17b-4a61-b95b-13330eda3101\") "
Nov 03 00:27:30.017055 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.017001    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-tmp-dir" pod "40799d7b-f17b-4a61-b95b-13330eda3101" (UID: "40799d7b-f17b-4a61-b95b-13330eda3101"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:30.017169 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.017067    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-home" pod "40799d7b-f17b-4a61-b95b-13330eda3101" (UID: "40799d7b-f17b-4a61-b95b-13330eda3101"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:30.017356 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.017334    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:27:30.017356 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.017356    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:27:30.017520 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.017369    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:27:30.019171 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.019140    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40799d7b-f17b-4a61-b95b-13330eda3101-tls-certs" pod "40799d7b-f17b-4a61-b95b-13330eda3101" (UID: "40799d7b-f17b-4a61-b95b-13330eda3101"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:27:30.019295 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.019193    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-dshm" pod "40799d7b-f17b-4a61-b95b-13330eda3101" (UID: "40799d7b-f17b-4a61-b95b-13330eda3101"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:30.019506 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.019480    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40799d7b-f17b-4a61-b95b-13330eda3101-kube-api-access-9pnbs" pod "40799d7b-f17b-4a61-b95b-13330eda3101" (UID: "40799d7b-f17b-4a61-b95b-13330eda3101"). InnerVolumeSpecName "kube-api-access-9pnbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:30.118445 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.118414    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40799d7b-f17b-4a61-b95b-13330eda3101-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:27:30.118445 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.118440    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40799d7b-f17b-4a61-b95b-13330eda3101-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:27:30.118445 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.118451    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pnbs\" (UniqueName: \"kubernetes.io/projected/40799d7b-f17b-4a61-b95b-13330eda3101-kube-api-access-9pnbs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:27:30.634695 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.634662    4294 generic.go:356] "Generic (PLEG): container finished" podID="40799d7b-f17b-4a61-b95b-13330eda3101" containerID="e378232d96dae09f9eeb6947f62aa81f2c1292da9a0475922d73bdd30dcc3b58" exitCode=0
Nov 03 00:27:30.634878 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.634708    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"
Nov 03 00:27:30.634878 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.634757    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w" event={"ID":"40799d7b-f17b-4a61-b95b-13330eda3101","Type":"ContainerDied","Data":"e378232d96dae09f9eeb6947f62aa81f2c1292da9a0475922d73bdd30dcc3b58"}
Nov 03 00:27:30.634878 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.634801    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w" event={"ID":"40799d7b-f17b-4a61-b95b-13330eda3101","Type":"ContainerDied","Data":"e6504f68f6190666ec13d42084a1ed88cfae4e35eeb1594b1adcbc4c61707bc7"}
Nov 03 00:27:30.634878 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.634819    4294 scope.go:122] "RemoveContainer" containerID="e378232d96dae09f9eeb6947f62aa81f2c1292da9a0475922d73bdd30dcc3b58"
Nov 03 00:27:30.644315 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.644286    4294 scope.go:122] "RemoveContainer" containerID="e378232d96dae09f9eeb6947f62aa81f2c1292da9a0475922d73bdd30dcc3b58"
Nov 03 00:27:30.644581 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:27:30.644562    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e378232d96dae09f9eeb6947f62aa81f2c1292da9a0475922d73bdd30dcc3b58\": container with ID starting with e378232d96dae09f9eeb6947f62aa81f2c1292da9a0475922d73bdd30dcc3b58 not found: ID does not exist" containerID="e378232d96dae09f9eeb6947f62aa81f2c1292da9a0475922d73bdd30dcc3b58"
Nov 03 00:27:30.644675 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.644589    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e378232d96dae09f9eeb6947f62aa81f2c1292da9a0475922d73bdd30dcc3b58"} err="failed to get container status \"e378232d96dae09f9eeb6947f62aa81f2c1292da9a0475922d73bdd30dcc3b58\": rpc error: code = NotFound desc = could not find container \"e378232d96dae09f9eeb6947f62aa81f2c1292da9a0475922d73bdd30dcc3b58\": container with ID starting with e378232d96dae09f9eeb6947f62aa81f2c1292da9a0475922d73bdd30dcc3b58 not found: ID does not exist"
Nov 03 00:27:30.792291 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.792261    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"]
Nov 03 00:27:30.839607 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:30.839537    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-adc00e/simulator-single-main-adc00e-kserve-8dbdd7875-nl25w"]
Nov 03 00:27:32.546453 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:32.546421    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40799d7b-f17b-4a61-b95b-13330eda3101" path="/var/lib/kubelet/pods/40799d7b-f17b-4a61-b95b-13330eda3101/volumes"
Nov 03 00:27:41.747058 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:27:41.747025    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:27:51.466414 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:51.466379    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:52.104552 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:52.104516    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:53.468111 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:53.468064    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:53.811322 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:53.811282    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:54.075995 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:54.075903    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:57.517614 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:57.517575    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:57.686268 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:27:57.686230    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:11.747751 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:28:11.747716    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:28:12.301902 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.301871    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"]
Nov 03 00:28:12.302302 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.302289    4294 cpu_manager.go:414] "containerMap: removing container" podUID="40799d7b-f17b-4a61-b95b-13330eda3101" containerName="main"
Nov 03 00:28:12.302355 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.302304    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40799d7b-f17b-4a61-b95b-13330eda3101" containerName="main"
Nov 03 00:28:12.302397 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.302368    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="40799d7b-f17b-4a61-b95b-13330eda3101" containerName="main"
Nov 03 00:28:12.305257 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.305242    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:12.382180 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.382149    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-07079f68-kserve-self-signed-certs\""
Nov 03 00:28:12.407573 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.407542    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"]
Nov 03 00:28:12.411376 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.411353    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kwxz\" (UniqueName: \"kubernetes.io/projected/a2831f02-3968-4db1-9857-d95924d94bf6-kube-api-access-7kwxz\") pod \"e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") " pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:12.411496 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.411474    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-tmp-dir\") pod \"e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") " pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:12.411551 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.411504    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2831f02-3968-4db1-9857-d95924d94bf6-tls-certs\") pod \"e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") " pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:12.411609 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.411572    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-home\") pod \"e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") " pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:12.411651 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.411624    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-model-cache\") pod \"e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") " pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:12.411699 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.411683    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-dshm\") pod \"e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") " pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:12.440681 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.440644    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:12.512803 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.512771    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kwxz\" (UniqueName: \"kubernetes.io/projected/a2831f02-3968-4db1-9857-d95924d94bf6-kube-api-access-7kwxz\") pod \"e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") " pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:12.513029 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.512847    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-tmp-dir\") pod \"e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") " pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:12.513029 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.512887    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2831f02-3968-4db1-9857-d95924d94bf6-tls-certs\") pod \"e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") " pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:12.513029 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.512915    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-home\") pod \"e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") " pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:12.513029 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.512936    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-model-cache\") pod \"e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") " pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:12.513029 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.512964    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-dshm\") pod \"e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") " pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:12.513495 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.513469    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-model-cache\") pod \"e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") " pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:12.513646 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.513467    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-home\") pod \"e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") " pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:12.513646 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.513518    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-tmp-dir\") pod \"e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") " pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:12.515779 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.515755    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-dshm\") pod \"e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") " pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:12.516181 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.516161    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2831f02-3968-4db1-9857-d95924d94bf6-tls-certs\") pod \"e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") " pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:12.567699 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.567625    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kwxz\" (UniqueName: \"kubernetes.io/projected/a2831f02-3968-4db1-9857-d95924d94bf6-kube-api-access-7kwxz\") pod \"e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") " pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:12.616779 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.616750    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:12.983024 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:12.982993    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"]
Nov 03 00:28:12.984146 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:28:12.984107    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2831f02_3968_4db1_9857_d95924d94bf6.slice/crio-6375182f351db38214178aa58d1998be271b35001b22bb03fe3bba385e454e00 WatchSource:0}: Error finding container 6375182f351db38214178aa58d1998be271b35001b22bb03fe3bba385e454e00: Status 404 returned error can't find the container with id 6375182f351db38214178aa58d1998be271b35001b22bb03fe3bba385e454e00
Nov 03 00:28:13.842141 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:13.842101    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj" event={"ID":"a2831f02-3968-4db1-9857-d95924d94bf6","Type":"ContainerStarted","Data":"23e5a812866c8732fc0b856e2b6820c3eddfe042e55003763aeb448d6f908d69"}
Nov 03 00:28:13.842141 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:13.842146    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj" event={"ID":"a2831f02-3968-4db1-9857-d95924d94bf6","Type":"ContainerStarted","Data":"6375182f351db38214178aa58d1998be271b35001b22bb03fe3bba385e454e00"}
Nov 03 00:28:13.953369 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:13.953315    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj" podStartSLOduration=1.95330067 podStartE2EDuration="1.95330067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:13.951154887 +0000 UTC m=+5733.519509737" watchObservedRunningTime="2026-11-03 00:28:13.95330067 +0000 UTC m=+5733.521655457"
Nov 03 00:28:14.616992 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:14.616947    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:19.771001 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:19.768057    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:28:19.771001 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:19.768118    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w" podUID="f60ec698-0068-4422-aff1-c8ecbe35f709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:28:22.617514 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:22.617480    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:22.629851 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:22.629825    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:22.952600 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:22.952525    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:28:38.070833 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.070802    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"]
Nov 03 00:28:38.074338 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.074306    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:38.127470 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.127439    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"]
Nov 03 00:28:38.172445 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.172400    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-07079f68-kserve-self-signed-certs\""
Nov 03 00:28:38.177826 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.177801    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qswt\" (UniqueName: \"kubernetes.io/projected/11f29c87-1b5b-40e5-aa2f-a06829bf055b-kube-api-access-4qswt\") pod \"e2e-same-id-b-07079f68-kserve-747cc59889-26frq\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") " pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:38.177961 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.177853    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-home\") pod \"e2e-same-id-b-07079f68-kserve-747cc59889-26frq\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") " pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:38.178036 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.177961    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-model-cache\") pod \"e2e-same-id-b-07079f68-kserve-747cc59889-26frq\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") " pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:38.178036 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.178009    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11f29c87-1b5b-40e5-aa2f-a06829bf055b-tls-certs\") pod \"e2e-same-id-b-07079f68-kserve-747cc59889-26frq\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") " pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:38.178115 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.178081    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-dshm\") pod \"e2e-same-id-b-07079f68-kserve-747cc59889-26frq\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") " pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:38.178150 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.178126    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-tmp-dir\") pod \"e2e-same-id-b-07079f68-kserve-747cc59889-26frq\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") " pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:38.211069 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.211041    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:38.279143 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.279111    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qswt\" (UniqueName: \"kubernetes.io/projected/11f29c87-1b5b-40e5-aa2f-a06829bf055b-kube-api-access-4qswt\") pod \"e2e-same-id-b-07079f68-kserve-747cc59889-26frq\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") " pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:38.279360 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.279158    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-home\") pod \"e2e-same-id-b-07079f68-kserve-747cc59889-26frq\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") " pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:38.279360 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.279192    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-model-cache\") pod \"e2e-same-id-b-07079f68-kserve-747cc59889-26frq\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") " pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:38.279360 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.279206    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11f29c87-1b5b-40e5-aa2f-a06829bf055b-tls-certs\") pod \"e2e-same-id-b-07079f68-kserve-747cc59889-26frq\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") " pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:38.279360 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.279227    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-dshm\") pod \"e2e-same-id-b-07079f68-kserve-747cc59889-26frq\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") " pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:38.279360 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.279259    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-tmp-dir\") pod \"e2e-same-id-b-07079f68-kserve-747cc59889-26frq\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") " pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:38.279650 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.279585    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-home\") pod \"e2e-same-id-b-07079f68-kserve-747cc59889-26frq\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") " pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:38.279706 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.279651    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-model-cache\") pod \"e2e-same-id-b-07079f68-kserve-747cc59889-26frq\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") " pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:38.279706 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.279677    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-tmp-dir\") pod \"e2e-same-id-b-07079f68-kserve-747cc59889-26frq\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") " pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:38.281479 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.281458    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-dshm\") pod \"e2e-same-id-b-07079f68-kserve-747cc59889-26frq\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") " pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:38.281897 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.281876    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11f29c87-1b5b-40e5-aa2f-a06829bf055b-tls-certs\") pod \"e2e-same-id-b-07079f68-kserve-747cc59889-26frq\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") " pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:38.333795 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.333734    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qswt\" (UniqueName: \"kubernetes.io/projected/11f29c87-1b5b-40e5-aa2f-a06829bf055b-kube-api-access-4qswt\") pod \"e2e-same-id-b-07079f68-kserve-747cc59889-26frq\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") " pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:38.386700 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.386674    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:38.750614 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:38.750590    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"]
Nov 03 00:28:38.751471 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:28:38.751448    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11f29c87_1b5b_40e5_aa2f_a06829bf055b.slice/crio-51ec8f3945db0057bdab4c2219b417b8a44a14277a7b494ffca256daab5b2820 WatchSource:0}: Error finding container 51ec8f3945db0057bdab4c2219b417b8a44a14277a7b494ffca256daab5b2820: Status 404 returned error can't find the container with id 51ec8f3945db0057bdab4c2219b417b8a44a14277a7b494ffca256daab5b2820
Nov 03 00:28:39.015677 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:39.015589    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq" event={"ID":"11f29c87-1b5b-40e5-aa2f-a06829bf055b","Type":"ContainerStarted","Data":"aba6232e9f39dbdb7a5b7d4d4273337d03063451718bb6177e647fc170724f7a"}
Nov 03 00:28:39.015677 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:39.015637    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq" event={"ID":"11f29c87-1b5b-40e5-aa2f-a06829bf055b","Type":"ContainerStarted","Data":"51ec8f3945db0057bdab4c2219b417b8a44a14277a7b494ffca256daab5b2820"}
Nov 03 00:28:39.126637 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:39.126578    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq" podStartSLOduration=1.126560715 podStartE2EDuration="1.126560715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:39.123779175 +0000 UTC m=+5758.692133963" watchObservedRunningTime="2026-11-03 00:28:39.126560715 +0000 UTC m=+5758.694915506"
Nov 03 00:28:39.386860 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:39.386790    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:41.751077 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:28:41.751034    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:28:48.386950 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:48.386913    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:48.399138 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:48.399114    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:49.072648 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:49.072617    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:28:54.138299 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:28:54.138265    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:11.751662 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:29:11.751629    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:29:20.300236 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:20.300205    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"]
Nov 03 00:29:20.361881 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:20.361854    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:22.949366 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:22.949336    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"]
Nov 03 00:29:22.998554 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:22.998519    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:35.319361 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:35.319313    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj" podUID="a2831f02-3968-4db1-9857-d95924d94bf6" containerName="main" containerID="cri-o://23e5a812866c8732fc0b856e2b6820c3eddfe042e55003763aeb448d6f908d69" gracePeriod=45
Nov 03 00:29:35.566998 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:35.566952    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:29:35.649439 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:35.649410    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-dshm\") pod \"a2831f02-3968-4db1-9857-d95924d94bf6\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") "
Nov 03 00:29:35.649591 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:35.649447    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2831f02-3968-4db1-9857-d95924d94bf6-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2831f02-3968-4db1-9857-d95924d94bf6-tls-certs\") pod \"a2831f02-3968-4db1-9857-d95924d94bf6\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") "
Nov 03 00:29:35.649591 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:35.649493    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2831f02-3968-4db1-9857-d95924d94bf6-kube-api-access-7kwxz\" (UniqueName: \"kubernetes.io/projected/a2831f02-3968-4db1-9857-d95924d94bf6-kube-api-access-7kwxz\") pod \"a2831f02-3968-4db1-9857-d95924d94bf6\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") "
Nov 03 00:29:35.649591 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:35.649515    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-tmp-dir\") pod \"a2831f02-3968-4db1-9857-d95924d94bf6\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") "
Nov 03 00:29:35.649591 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:35.649550    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-model-cache\") pod \"a2831f02-3968-4db1-9857-d95924d94bf6\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") "
Nov 03 00:29:35.649827 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:35.649664    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-home\" (UniqueName: \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-home\") pod \"a2831f02-3968-4db1-9857-d95924d94bf6\" (UID: \"a2831f02-3968-4db1-9857-d95924d94bf6\") "
Nov 03 00:29:35.649893 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:35.649820    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-tmp-dir" pod "a2831f02-3968-4db1-9857-d95924d94bf6" (UID: "a2831f02-3968-4db1-9857-d95924d94bf6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:29:35.649893 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:35.649864    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-model-cache" pod "a2831f02-3968-4db1-9857-d95924d94bf6" (UID: "a2831f02-3968-4db1-9857-d95924d94bf6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:29:35.650024 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:35.649902    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-home" pod "a2831f02-3968-4db1-9857-d95924d94bf6" (UID: "a2831f02-3968-4db1-9857-d95924d94bf6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:29:35.650114 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:35.650094    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:29:35.650169 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:35.650120    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:29:35.650169 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:35.650136    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:29:35.651599 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:35.651574    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-dshm" pod "a2831f02-3968-4db1-9857-d95924d94bf6" (UID: "a2831f02-3968-4db1-9857-d95924d94bf6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:29:35.651718 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:35.651610    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2831f02-3968-4db1-9857-d95924d94bf6-kube-api-access-7kwxz" pod "a2831f02-3968-4db1-9857-d95924d94bf6" (UID: "a2831f02-3968-4db1-9857-d95924d94bf6"). InnerVolumeSpecName "kube-api-access-7kwxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:29:35.651718 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:35.651667    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2831f02-3968-4db1-9857-d95924d94bf6-tls-certs" pod "a2831f02-3968-4db1-9857-d95924d94bf6" (UID: "a2831f02-3968-4db1-9857-d95924d94bf6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:29:35.751192 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:35.751169    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2831f02-3968-4db1-9857-d95924d94bf6-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:29:35.751192 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:35.751189    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2831f02-3968-4db1-9857-d95924d94bf6-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:29:35.751378 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:35.751199    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kwxz\" (UniqueName: \"kubernetes.io/projected/a2831f02-3968-4db1-9857-d95924d94bf6-kube-api-access-7kwxz\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:29:36.343250 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:36.343216    4294 generic.go:356] "Generic (PLEG): container finished" podID="a2831f02-3968-4db1-9857-d95924d94bf6" containerID="23e5a812866c8732fc0b856e2b6820c3eddfe042e55003763aeb448d6f908d69" exitCode=0
Nov 03 00:29:36.343723 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:36.343261    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"
Nov 03 00:29:36.343723 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:36.343300    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj" event={"ID":"a2831f02-3968-4db1-9857-d95924d94bf6","Type":"ContainerDied","Data":"23e5a812866c8732fc0b856e2b6820c3eddfe042e55003763aeb448d6f908d69"}
Nov 03 00:29:36.343723 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:36.343348    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj" event={"ID":"a2831f02-3968-4db1-9857-d95924d94bf6","Type":"ContainerDied","Data":"6375182f351db38214178aa58d1998be271b35001b22bb03fe3bba385e454e00"}
Nov 03 00:29:36.343723 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:36.343365    4294 scope.go:122] "RemoveContainer" containerID="23e5a812866c8732fc0b856e2b6820c3eddfe042e55003763aeb448d6f908d69"
Nov 03 00:29:36.352557 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:36.352541    4294 scope.go:122] "RemoveContainer" containerID="23e5a812866c8732fc0b856e2b6820c3eddfe042e55003763aeb448d6f908d69"
Nov 03 00:29:36.352796 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:29:36.352777    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23e5a812866c8732fc0b856e2b6820c3eddfe042e55003763aeb448d6f908d69\": container with ID starting with 23e5a812866c8732fc0b856e2b6820c3eddfe042e55003763aeb448d6f908d69 not found: ID does not exist" containerID="23e5a812866c8732fc0b856e2b6820c3eddfe042e55003763aeb448d6f908d69"
Nov 03 00:29:36.352869 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:36.352805    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23e5a812866c8732fc0b856e2b6820c3eddfe042e55003763aeb448d6f908d69"} err="failed to get container status \"23e5a812866c8732fc0b856e2b6820c3eddfe042e55003763aeb448d6f908d69\": rpc error: code = NotFound desc = could not find container \"23e5a812866c8732fc0b856e2b6820c3eddfe042e55003763aeb448d6f908d69\": container with ID starting with 23e5a812866c8732fc0b856e2b6820c3eddfe042e55003763aeb448d6f908d69 not found: ID does not exist"
Nov 03 00:29:36.500971 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:36.500939    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"]
Nov 03 00:29:36.555150 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:36.549498    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-07079f68-kserve-6d7cf78565-zmfqj"]
Nov 03 00:29:37.968974 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:37.968918    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq" podUID="11f29c87-1b5b-40e5-aa2f-a06829bf055b" containerName="main" containerID="cri-o://aba6232e9f39dbdb7a5b7d4d4273337d03063451718bb6177e647fc170724f7a" gracePeriod=45
Nov 03 00:29:38.063039 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.062999    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq" podUID="11f29c87-1b5b-40e5-aa2f-a06829bf055b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.221:8000/ready\": dial tcp 10.133.0.221:8000: connect: connection refused"
Nov 03 00:29:38.223127 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.223071    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:29:38.273203 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.273174    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f29c87-1b5b-40e5-aa2f-a06829bf055b-kube-api-access-4qswt\" (UniqueName: \"kubernetes.io/projected/11f29c87-1b5b-40e5-aa2f-a06829bf055b-kube-api-access-4qswt\") pod \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") "
Nov 03 00:29:38.273387 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.273243    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-home\" (UniqueName: \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-home\") pod \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") "
Nov 03 00:29:38.273387 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.273269    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f29c87-1b5b-40e5-aa2f-a06829bf055b-tls-certs\" (UniqueName: \"kubernetes.io/secret/11f29c87-1b5b-40e5-aa2f-a06829bf055b-tls-certs\") pod \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") "
Nov 03 00:29:38.273387 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.273294    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-tmp-dir\") pod \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") "
Nov 03 00:29:38.273387 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.273320    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-model-cache\") pod \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") "
Nov 03 00:29:38.273387 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.273378    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-dshm\") pod \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\" (UID: \"11f29c87-1b5b-40e5-aa2f-a06829bf055b\") "
Nov 03 00:29:38.273662 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.273587    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-home" pod "11f29c87-1b5b-40e5-aa2f-a06829bf055b" (UID: "11f29c87-1b5b-40e5-aa2f-a06829bf055b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:29:38.273662 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.273575    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-tmp-dir" pod "11f29c87-1b5b-40e5-aa2f-a06829bf055b" (UID: "11f29c87-1b5b-40e5-aa2f-a06829bf055b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:29:38.273746 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.273667    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-model-cache" pod "11f29c87-1b5b-40e5-aa2f-a06829bf055b" (UID: "11f29c87-1b5b-40e5-aa2f-a06829bf055b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:29:38.275356 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.275319    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f29c87-1b5b-40e5-aa2f-a06829bf055b-kube-api-access-4qswt" pod "11f29c87-1b5b-40e5-aa2f-a06829bf055b" (UID: "11f29c87-1b5b-40e5-aa2f-a06829bf055b"). InnerVolumeSpecName "kube-api-access-4qswt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:29:38.275458 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.275416    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f29c87-1b5b-40e5-aa2f-a06829bf055b-tls-certs" pod "11f29c87-1b5b-40e5-aa2f-a06829bf055b" (UID: "11f29c87-1b5b-40e5-aa2f-a06829bf055b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:29:38.275504 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.275463    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-dshm" pod "11f29c87-1b5b-40e5-aa2f-a06829bf055b" (UID: "11f29c87-1b5b-40e5-aa2f-a06829bf055b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:29:38.357696 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.357657    4294 generic.go:356] "Generic (PLEG): container finished" podID="11f29c87-1b5b-40e5-aa2f-a06829bf055b" containerID="aba6232e9f39dbdb7a5b7d4d4273337d03063451718bb6177e647fc170724f7a" exitCode=0
Nov 03 00:29:38.357880 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.357709    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"
Nov 03 00:29:38.357880 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.357743    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq" event={"ID":"11f29c87-1b5b-40e5-aa2f-a06829bf055b","Type":"ContainerDied","Data":"aba6232e9f39dbdb7a5b7d4d4273337d03063451718bb6177e647fc170724f7a"}
Nov 03 00:29:38.357880 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.357794    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq" event={"ID":"11f29c87-1b5b-40e5-aa2f-a06829bf055b","Type":"ContainerDied","Data":"51ec8f3945db0057bdab4c2219b417b8a44a14277a7b494ffca256daab5b2820"}
Nov 03 00:29:38.357880 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.357816    4294 scope.go:122] "RemoveContainer" containerID="aba6232e9f39dbdb7a5b7d4d4273337d03063451718bb6177e647fc170724f7a"
Nov 03 00:29:38.368208 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.368184    4294 scope.go:122] "RemoveContainer" containerID="aba6232e9f39dbdb7a5b7d4d4273337d03063451718bb6177e647fc170724f7a"
Nov 03 00:29:38.368473 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:29:38.368457    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aba6232e9f39dbdb7a5b7d4d4273337d03063451718bb6177e647fc170724f7a\": container with ID starting with aba6232e9f39dbdb7a5b7d4d4273337d03063451718bb6177e647fc170724f7a not found: ID does not exist" containerID="aba6232e9f39dbdb7a5b7d4d4273337d03063451718bb6177e647fc170724f7a"
Nov 03 00:29:38.368539 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.368486    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba6232e9f39dbdb7a5b7d4d4273337d03063451718bb6177e647fc170724f7a"} err="failed to get container status \"aba6232e9f39dbdb7a5b7d4d4273337d03063451718bb6177e647fc170724f7a\": rpc error: code = NotFound desc = could not find container \"aba6232e9f39dbdb7a5b7d4d4273337d03063451718bb6177e647fc170724f7a\": container with ID starting with aba6232e9f39dbdb7a5b7d4d4273337d03063451718bb6177e647fc170724f7a not found: ID does not exist"
Nov 03 00:29:38.374059 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.374039    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qswt\" (UniqueName: \"kubernetes.io/projected/11f29c87-1b5b-40e5-aa2f-a06829bf055b-kube-api-access-4qswt\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:29:38.374159 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.374061    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:29:38.374159 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.374074    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11f29c87-1b5b-40e5-aa2f-a06829bf055b-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:29:38.374159 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.374086    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:29:38.374159 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.374099    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:29:38.374159 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.374114    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11f29c87-1b5b-40e5-aa2f-a06829bf055b-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:29:38.513658 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.513628    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"]
Nov 03 00:29:38.517166 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.517144    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-07079f68-kserve-747cc59889-26frq"]
Nov 03 00:29:38.546529 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.546497    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f29c87-1b5b-40e5-aa2f-a06829bf055b" path="/var/lib/kubelet/pods/11f29c87-1b5b-40e5-aa2f-a06829bf055b/volumes"
Nov 03 00:29:38.547024 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:38.547005    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2831f02-3968-4db1-9857-d95924d94bf6" path="/var/lib/kubelet/pods/a2831f02-3968-4db1-9857-d95924d94bf6/volumes"
Nov 03 00:29:41.751822 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:29:41.751786    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:29:54.468320 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:29:54.468279    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:04.191910 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:04.191871    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:08.029932 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:08.029897    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59bd66d49-ddzr5"]
Nov 03 00:30:08.030361 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:08.030323    4294 cpu_manager.go:414] "containerMap: removing container" podUID="a2831f02-3968-4db1-9857-d95924d94bf6" containerName="main"
Nov 03 00:30:08.030361 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:08.030334    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2831f02-3968-4db1-9857-d95924d94bf6" containerName="main"
Nov 03 00:30:08.030361 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:08.030349    4294 cpu_manager.go:414] "containerMap: removing container" podUID="11f29c87-1b5b-40e5-aa2f-a06829bf055b" containerName="main"
Nov 03 00:30:08.030361 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:08.030354    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f29c87-1b5b-40e5-aa2f-a06829bf055b" containerName="main"
Nov 03 00:30:08.030510 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:08.030423    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2831f02-3968-4db1-9857-d95924d94bf6" containerName="main"
Nov 03 00:30:08.030510 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:08.030432    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f29c87-1b5b-40e5-aa2f-a06829bf055b" containerName="main"
Nov 03 00:30:08.033352 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:08.033336    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59bd66d49-ddzr5"
Nov 03 00:30:08.086584 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:08.086557    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59bd66d49-ddzr5"]
Nov 03 00:30:08.102475 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:08.102440    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f438f93-ef32-4dd5-9185-76f7a3a205ab-plugins-config-volume\") pod \"payload-pre-processing-59bd66d49-ddzr5\" (UID: \"6f438f93-ef32-4dd5-9185-76f7a3a205ab\") " pod="openshift-ingress/payload-pre-processing-59bd66d49-ddzr5"
Nov 03 00:30:08.102475 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:08.102474    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5t9n\" (UniqueName: \"kubernetes.io/projected/6f438f93-ef32-4dd5-9185-76f7a3a205ab-kube-api-access-c5t9n\") pod \"payload-pre-processing-59bd66d49-ddzr5\" (UID: \"6f438f93-ef32-4dd5-9185-76f7a3a205ab\") " pod="openshift-ingress/payload-pre-processing-59bd66d49-ddzr5"
Nov 03 00:30:08.203482 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:08.203445    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f438f93-ef32-4dd5-9185-76f7a3a205ab-plugins-config-volume\") pod \"payload-pre-processing-59bd66d49-ddzr5\" (UID: \"6f438f93-ef32-4dd5-9185-76f7a3a205ab\") " pod="openshift-ingress/payload-pre-processing-59bd66d49-ddzr5"
Nov 03 00:30:08.203482 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:08.203484    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5t9n\" (UniqueName: \"kubernetes.io/projected/6f438f93-ef32-4dd5-9185-76f7a3a205ab-kube-api-access-c5t9n\") pod \"payload-pre-processing-59bd66d49-ddzr5\" (UID: \"6f438f93-ef32-4dd5-9185-76f7a3a205ab\") " pod="openshift-ingress/payload-pre-processing-59bd66d49-ddzr5"
Nov 03 00:30:08.204160 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:08.204140    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f438f93-ef32-4dd5-9185-76f7a3a205ab-plugins-config-volume\") pod \"payload-pre-processing-59bd66d49-ddzr5\" (UID: \"6f438f93-ef32-4dd5-9185-76f7a3a205ab\") " pod="openshift-ingress/payload-pre-processing-59bd66d49-ddzr5"
Nov 03 00:30:08.258158 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:08.258128    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5t9n\" (UniqueName: \"kubernetes.io/projected/6f438f93-ef32-4dd5-9185-76f7a3a205ab-kube-api-access-c5t9n\") pod \"payload-pre-processing-59bd66d49-ddzr5\" (UID: \"6f438f93-ef32-4dd5-9185-76f7a3a205ab\") " pod="openshift-ingress/payload-pre-processing-59bd66d49-ddzr5"
Nov 03 00:30:08.345935 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:08.345871    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59bd66d49-ddzr5"
Nov 03 00:30:08.710287 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:08.710262    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59bd66d49-ddzr5"]
Nov 03 00:30:08.711972 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:30:08.711945    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f438f93_ef32_4dd5_9185_76f7a3a205ab.slice/crio-163a4362307e04372a0e23c8405ab7009bab5dcfbb8b2cf148a471b97cb0028b WatchSource:0}: Error finding container 163a4362307e04372a0e23c8405ab7009bab5dcfbb8b2cf148a471b97cb0028b: Status 404 returned error can't find the container with id 163a4362307e04372a0e23c8405ab7009bab5dcfbb8b2cf148a471b97cb0028b
Nov 03 00:30:09.630671 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:09.630626    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59bd66d49-ddzr5" event={"ID":"6f438f93-ef32-4dd5-9185-76f7a3a205ab","Type":"ContainerStarted","Data":"4e587996bd5e00d28299fe33e30c2384baed51bd52ea6c2f6bb4e80e6477919f"}
Nov 03 00:30:09.631111 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:09.630688    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59bd66d49-ddzr5" event={"ID":"6f438f93-ef32-4dd5-9185-76f7a3a205ab","Type":"ContainerStarted","Data":"163a4362307e04372a0e23c8405ab7009bab5dcfbb8b2cf148a471b97cb0028b"}
Nov 03 00:30:09.631111 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:09.630761    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59bd66d49-ddzr5"
Nov 03 00:30:09.744678 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:09.744615    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59bd66d49-ddzr5" podStartSLOduration=2.744596811 podStartE2EDuration="2.744596811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:09.740879378 +0000 UTC m=+5849.309234167" watchObservedRunningTime="2026-11-03 00:30:09.744596811 +0000 UTC m=+5849.312951600"
Nov 03 00:30:11.747900 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:30:11.747866    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:30:16.260295 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:16.260263    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:16.430025 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:16.429975    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:20.638304 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:20.638272    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59bd66d49-ddzr5"
Nov 03 00:30:20.775023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:20.774972    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l"]
Nov 03 00:30:22.784190 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:22.784159    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-78847578d5-wm8k4"]
Nov 03 00:30:22.794327 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:22.794307    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-78847578d5-wm8k4"
Nov 03 00:30:22.841391 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:22.841363    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-78847578d5-wm8k4"]
Nov 03 00:30:22.940648 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:22.940612    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5wrm\" (UniqueName: \"kubernetes.io/projected/91945b75-548c-46b6-a76c-bde3e6afcb07-kube-api-access-c5wrm\") pod \"payload-processing-78847578d5-wm8k4\" (UID: \"91945b75-548c-46b6-a76c-bde3e6afcb07\") " pod="openshift-ingress/payload-processing-78847578d5-wm8k4"
Nov 03 00:30:22.940833 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:22.940756    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91945b75-548c-46b6-a76c-bde3e6afcb07-plugins-config-volume\") pod \"payload-processing-78847578d5-wm8k4\" (UID: \"91945b75-548c-46b6-a76c-bde3e6afcb07\") " pod="openshift-ingress/payload-processing-78847578d5-wm8k4"
Nov 03 00:30:23.041453 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:23.041377    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5wrm\" (UniqueName: \"kubernetes.io/projected/91945b75-548c-46b6-a76c-bde3e6afcb07-kube-api-access-c5wrm\") pod \"payload-processing-78847578d5-wm8k4\" (UID: \"91945b75-548c-46b6-a76c-bde3e6afcb07\") " pod="openshift-ingress/payload-processing-78847578d5-wm8k4"
Nov 03 00:30:23.041601 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:23.041472    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91945b75-548c-46b6-a76c-bde3e6afcb07-plugins-config-volume\") pod \"payload-processing-78847578d5-wm8k4\" (UID: \"91945b75-548c-46b6-a76c-bde3e6afcb07\") " pod="openshift-ingress/payload-processing-78847578d5-wm8k4"
Nov 03 00:30:23.042194 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:23.042172    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91945b75-548c-46b6-a76c-bde3e6afcb07-plugins-config-volume\") pod \"payload-processing-78847578d5-wm8k4\" (UID: \"91945b75-548c-46b6-a76c-bde3e6afcb07\") " pod="openshift-ingress/payload-processing-78847578d5-wm8k4"
Nov 03 00:30:23.097015 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:23.096968    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5wrm\" (UniqueName: \"kubernetes.io/projected/91945b75-548c-46b6-a76c-bde3e6afcb07-kube-api-access-c5wrm\") pod \"payload-processing-78847578d5-wm8k4\" (UID: \"91945b75-548c-46b6-a76c-bde3e6afcb07\") " pod="openshift-ingress/payload-processing-78847578d5-wm8k4"
Nov 03 00:30:23.106849 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:23.106828    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-78847578d5-wm8k4"
Nov 03 00:30:23.470591 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:23.470566    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-78847578d5-wm8k4"]
Nov 03 00:30:23.471756 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:30:23.471730    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91945b75_548c_46b6_a76c_bde3e6afcb07.slice/crio-e358d1ff37661bd60f1b814dc9d489546fdcb65f016f423104f623bcec9d4c1d WatchSource:0}: Error finding container e358d1ff37661bd60f1b814dc9d489546fdcb65f016f423104f623bcec9d4c1d: Status 404 returned error can't find the container with id e358d1ff37661bd60f1b814dc9d489546fdcb65f016f423104f623bcec9d4c1d
Nov 03 00:30:23.697013 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:23.696959    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-78847578d5-wm8k4" event={"ID":"91945b75-548c-46b6-a76c-bde3e6afcb07","Type":"ContainerStarted","Data":"bae9ec25ffeab0be3468a5c9d41925ad50c8961073bcdad1c98fea8e05eaad54"}
Nov 03 00:30:23.697209 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:23.697043    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-78847578d5-wm8k4" event={"ID":"91945b75-548c-46b6-a76c-bde3e6afcb07","Type":"ContainerStarted","Data":"e358d1ff37661bd60f1b814dc9d489546fdcb65f016f423104f623bcec9d4c1d"}
Nov 03 00:30:23.697209 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:23.697171    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-78847578d5-wm8k4"
Nov 03 00:30:23.811049 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:23.810974    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-78847578d5-wm8k4" podStartSLOduration=1.810956151 podStartE2EDuration="1.810956151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:23.808755558 +0000 UTC m=+5863.377110344" watchObservedRunningTime="2026-11-03 00:30:23.810956151 +0000 UTC m=+5863.379310940"
Nov 03 00:30:25.776470 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:25.776430    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l" podUID="939ab75a-b60a-408d-8661-1f4e17c4b809" containerName="payload-pre-processing" containerID="cri-o://b5a5b590b70fbd9d7e790edccc8e3da7366e7d90bd6d0cb21642a32fd806b653" gracePeriod=25
Nov 03 00:30:26.023292 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:26.023267    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l"
Nov 03 00:30:26.171242 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:26.171167    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/939ab75a-b60a-408d-8661-1f4e17c4b809-kube-api-access-v9x4j\" (UniqueName: \"kubernetes.io/projected/939ab75a-b60a-408d-8661-1f4e17c4b809-kube-api-access-v9x4j\") pod \"939ab75a-b60a-408d-8661-1f4e17c4b809\" (UID: \"939ab75a-b60a-408d-8661-1f4e17c4b809\") "
Nov 03 00:30:26.171398 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:26.171248    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/939ab75a-b60a-408d-8661-1f4e17c4b809-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939ab75a-b60a-408d-8661-1f4e17c4b809-plugins-config-volume\") pod \"939ab75a-b60a-408d-8661-1f4e17c4b809\" (UID: \"939ab75a-b60a-408d-8661-1f4e17c4b809\") "
Nov 03 00:30:26.171608 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:26.171586    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/939ab75a-b60a-408d-8661-1f4e17c4b809-plugins-config-volume" pod "939ab75a-b60a-408d-8661-1f4e17c4b809" (UID: "939ab75a-b60a-408d-8661-1f4e17c4b809"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:26.173449 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:26.173426    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/939ab75a-b60a-408d-8661-1f4e17c4b809-kube-api-access-v9x4j" pod "939ab75a-b60a-408d-8661-1f4e17c4b809" (UID: "939ab75a-b60a-408d-8661-1f4e17c4b809"). InnerVolumeSpecName "kube-api-access-v9x4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:26.272836 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:26.272807    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9x4j\" (UniqueName: \"kubernetes.io/projected/939ab75a-b60a-408d-8661-1f4e17c4b809-kube-api-access-v9x4j\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:30:26.272836 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:26.272834    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/939ab75a-b60a-408d-8661-1f4e17c4b809-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:30:26.711790 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:26.711759    4294 generic.go:356] "Generic (PLEG): container finished" podID="939ab75a-b60a-408d-8661-1f4e17c4b809" containerID="b5a5b590b70fbd9d7e790edccc8e3da7366e7d90bd6d0cb21642a32fd806b653" exitCode=0
Nov 03 00:30:26.711972 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:26.711804    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l"
Nov 03 00:30:26.711972 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:26.711834    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l" event={"ID":"939ab75a-b60a-408d-8661-1f4e17c4b809","Type":"ContainerDied","Data":"b5a5b590b70fbd9d7e790edccc8e3da7366e7d90bd6d0cb21642a32fd806b653"}
Nov 03 00:30:26.711972 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:26.711879    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l" event={"ID":"939ab75a-b60a-408d-8661-1f4e17c4b809","Type":"ContainerDied","Data":"c7fa4711aa05e1571aa43339158d221761cd8585cbfdf1ccdaec2f608eb53670"}
Nov 03 00:30:26.711972 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:26.711901    4294 scope.go:122] "RemoveContainer" containerID="b5a5b590b70fbd9d7e790edccc8e3da7366e7d90bd6d0cb21642a32fd806b653"
Nov 03 00:30:26.720359 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:26.720343    4294 scope.go:122] "RemoveContainer" containerID="b5a5b590b70fbd9d7e790edccc8e3da7366e7d90bd6d0cb21642a32fd806b653"
Nov 03 00:30:26.720628 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:30:26.720611    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5a5b590b70fbd9d7e790edccc8e3da7366e7d90bd6d0cb21642a32fd806b653\": container with ID starting with b5a5b590b70fbd9d7e790edccc8e3da7366e7d90bd6d0cb21642a32fd806b653 not found: ID does not exist" containerID="b5a5b590b70fbd9d7e790edccc8e3da7366e7d90bd6d0cb21642a32fd806b653"
Nov 03 00:30:26.720670 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:26.720635    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5a5b590b70fbd9d7e790edccc8e3da7366e7d90bd6d0cb21642a32fd806b653"} err="failed to get container status \"b5a5b590b70fbd9d7e790edccc8e3da7366e7d90bd6d0cb21642a32fd806b653\": rpc error: code = NotFound desc = could not find container \"b5a5b590b70fbd9d7e790edccc8e3da7366e7d90bd6d0cb21642a32fd806b653\": container with ID starting with b5a5b590b70fbd9d7e790edccc8e3da7366e7d90bd6d0cb21642a32fd806b653 not found: ID does not exist"
Nov 03 00:30:26.873751 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:26.873672    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l"]
Nov 03 00:30:26.921130 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:26.921099    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-c7k9l"]
Nov 03 00:30:28.546783 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:28.546749    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="939ab75a-b60a-408d-8661-1f4e17c4b809" path="/var/lib/kubelet/pods/939ab75a-b60a-408d-8661-1f4e17c4b809/volumes"
Nov 03 00:30:34.704260 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:34.704231    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-78847578d5-wm8k4"
Nov 03 00:30:34.863623 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:34.863584    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rnnmk"]
Nov 03 00:30:37.063229 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.063189    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-b45869bf8-wtj54"]
Nov 03 00:30:37.063831 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.063808    4294 cpu_manager.go:414] "containerMap: removing container" podUID="939ab75a-b60a-408d-8661-1f4e17c4b809" containerName="payload-pre-processing"
Nov 03 00:30:37.064021 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.063833    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="939ab75a-b60a-408d-8661-1f4e17c4b809" containerName="payload-pre-processing"
Nov 03 00:30:37.064021 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.063926    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="939ab75a-b60a-408d-8661-1f4e17c4b809" containerName="payload-pre-processing"
Nov 03 00:30:37.067293 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.067272    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-b45869bf8-wtj54"
Nov 03 00:30:37.119961 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.119931    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-b45869bf8-wtj54"]
Nov 03 00:30:37.173333 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.173299    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94f9d08-36a2-4efb-b95c-3798eb31d227-plugins-config-volume\") pod \"payload-processing-b45869bf8-wtj54\" (UID: \"e94f9d08-36a2-4efb-b95c-3798eb31d227\") " pod="openshift-ingress/payload-processing-b45869bf8-wtj54"
Nov 03 00:30:37.173482 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.173421    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzk4x\" (UniqueName: \"kubernetes.io/projected/e94f9d08-36a2-4efb-b95c-3798eb31d227-kube-api-access-qzk4x\") pod \"payload-processing-b45869bf8-wtj54\" (UID: \"e94f9d08-36a2-4efb-b95c-3798eb31d227\") " pod="openshift-ingress/payload-processing-b45869bf8-wtj54"
Nov 03 00:30:37.223099 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.223072    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-8499744ffb-z7wfd"]
Nov 03 00:30:37.226564 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.226549    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-8499744ffb-z7wfd"
Nov 03 00:30:37.274612 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.274585    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5fa5dc-4df9-41a3-95e0-55dd180cb73f-plugins-config-volume\") pod \"payload-pre-processing-8499744ffb-z7wfd\" (UID: \"8d5fa5dc-4df9-41a3-95e0-55dd180cb73f\") " pod="openshift-ingress/payload-pre-processing-8499744ffb-z7wfd"
Nov 03 00:30:37.274710 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.274634    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94f9d08-36a2-4efb-b95c-3798eb31d227-plugins-config-volume\") pod \"payload-processing-b45869bf8-wtj54\" (UID: \"e94f9d08-36a2-4efb-b95c-3798eb31d227\") " pod="openshift-ingress/payload-processing-b45869bf8-wtj54"
Nov 03 00:30:37.274710 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.274700    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4dzq\" (UniqueName: \"kubernetes.io/projected/8d5fa5dc-4df9-41a3-95e0-55dd180cb73f-kube-api-access-k4dzq\") pod \"payload-pre-processing-8499744ffb-z7wfd\" (UID: \"8d5fa5dc-4df9-41a3-95e0-55dd180cb73f\") " pod="openshift-ingress/payload-pre-processing-8499744ffb-z7wfd"
Nov 03 00:30:37.274784 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.274744    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzk4x\" (UniqueName: \"kubernetes.io/projected/e94f9d08-36a2-4efb-b95c-3798eb31d227-kube-api-access-qzk4x\") pod \"payload-processing-b45869bf8-wtj54\" (UID: \"e94f9d08-36a2-4efb-b95c-3798eb31d227\") " pod="openshift-ingress/payload-processing-b45869bf8-wtj54"
Nov 03 00:30:37.275305 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.275286    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94f9d08-36a2-4efb-b95c-3798eb31d227-plugins-config-volume\") pod \"payload-processing-b45869bf8-wtj54\" (UID: \"e94f9d08-36a2-4efb-b95c-3798eb31d227\") " pod="openshift-ingress/payload-processing-b45869bf8-wtj54"
Nov 03 00:30:37.279306 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.279283    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-8499744ffb-z7wfd"]
Nov 03 00:30:37.332131 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.332078    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzk4x\" (UniqueName: \"kubernetes.io/projected/e94f9d08-36a2-4efb-b95c-3798eb31d227-kube-api-access-qzk4x\") pod \"payload-processing-b45869bf8-wtj54\" (UID: \"e94f9d08-36a2-4efb-b95c-3798eb31d227\") " pod="openshift-ingress/payload-processing-b45869bf8-wtj54"
Nov 03 00:30:37.375740 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.375717    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5fa5dc-4df9-41a3-95e0-55dd180cb73f-plugins-config-volume\") pod \"payload-pre-processing-8499744ffb-z7wfd\" (UID: \"8d5fa5dc-4df9-41a3-95e0-55dd180cb73f\") " pod="openshift-ingress/payload-pre-processing-8499744ffb-z7wfd"
Nov 03 00:30:37.375831 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.375766    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4dzq\" (UniqueName: \"kubernetes.io/projected/8d5fa5dc-4df9-41a3-95e0-55dd180cb73f-kube-api-access-k4dzq\") pod \"payload-pre-processing-8499744ffb-z7wfd\" (UID: \"8d5fa5dc-4df9-41a3-95e0-55dd180cb73f\") " pod="openshift-ingress/payload-pre-processing-8499744ffb-z7wfd"
Nov 03 00:30:37.376377 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.376358    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d5fa5dc-4df9-41a3-95e0-55dd180cb73f-plugins-config-volume\") pod \"payload-pre-processing-8499744ffb-z7wfd\" (UID: \"8d5fa5dc-4df9-41a3-95e0-55dd180cb73f\") " pod="openshift-ingress/payload-pre-processing-8499744ffb-z7wfd"
Nov 03 00:30:37.379610 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.379594    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-b45869bf8-wtj54"
Nov 03 00:30:37.432891 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.432832    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4dzq\" (UniqueName: \"kubernetes.io/projected/8d5fa5dc-4df9-41a3-95e0-55dd180cb73f-kube-api-access-k4dzq\") pod \"payload-pre-processing-8499744ffb-z7wfd\" (UID: \"8d5fa5dc-4df9-41a3-95e0-55dd180cb73f\") " pod="openshift-ingress/payload-pre-processing-8499744ffb-z7wfd"
Nov 03 00:30:37.539030 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.538994    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-8499744ffb-z7wfd"
Nov 03 00:30:37.752119 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.752095    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-b45869bf8-wtj54"]
Nov 03 00:30:37.753328 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:30:37.753298    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94f9d08_36a2_4efb_b95c_3798eb31d227.slice/crio-956d55b3b106122dd8eb38a53761a434ca12c256d7fd53d573fb15df1f85ee84 WatchSource:0}: Error finding container 956d55b3b106122dd8eb38a53761a434ca12c256d7fd53d573fb15df1f85ee84: Status 404 returned error can't find the container with id 956d55b3b106122dd8eb38a53761a434ca12c256d7fd53d573fb15df1f85ee84
Nov 03 00:30:37.763737 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.763711    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-b45869bf8-wtj54" event={"ID":"e94f9d08-36a2-4efb-b95c-3798eb31d227","Type":"ContainerStarted","Data":"956d55b3b106122dd8eb38a53761a434ca12c256d7fd53d573fb15df1f85ee84"}
Nov 03 00:30:37.900462 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:37.900434    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-8499744ffb-z7wfd"]
Nov 03 00:30:37.901507 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:30:37.901483    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5fa5dc_4df9_41a3_95e0_55dd180cb73f.slice/crio-005b0f3c7b4074f1193e7deb6b894654ca6a7191ddb441c5c01e6cb478435632 WatchSource:0}: Error finding container 005b0f3c7b4074f1193e7deb6b894654ca6a7191ddb441c5c01e6cb478435632: Status 404 returned error can't find the container with id 005b0f3c7b4074f1193e7deb6b894654ca6a7191ddb441c5c01e6cb478435632
Nov 03 00:30:38.778338 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:38.778300    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-8499744ffb-z7wfd" event={"ID":"8d5fa5dc-4df9-41a3-95e0-55dd180cb73f","Type":"ContainerStarted","Data":"7572cd1f427ea735dc6609cf5cd738a642f0f336a26186fd8212ea42cf6e366a"}
Nov 03 00:30:38.778825 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:38.778381    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-8499744ffb-z7wfd" event={"ID":"8d5fa5dc-4df9-41a3-95e0-55dd180cb73f","Type":"ContainerStarted","Data":"005b0f3c7b4074f1193e7deb6b894654ca6a7191ddb441c5c01e6cb478435632"}
Nov 03 00:30:38.778825 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:38.778430    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-8499744ffb-z7wfd"
Nov 03 00:30:38.780023 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:38.779997    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-b45869bf8-wtj54" event={"ID":"e94f9d08-36a2-4efb-b95c-3798eb31d227","Type":"ContainerStarted","Data":"2d66fcca77cacaace598b997e4e99fcb0467bf2d73c6db6e6fddefc9f20b1d4c"}
Nov 03 00:30:38.780174 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:38.780076    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-b45869bf8-wtj54"
Nov 03 00:30:38.897565 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:38.897521    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-8499744ffb-z7wfd" podStartSLOduration=1.897509589 podStartE2EDuration="1.897509589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:38.896322228 +0000 UTC m=+5878.464677015" watchObservedRunningTime="2026-11-03 00:30:38.897509589 +0000 UTC m=+5878.465864375"
Nov 03 00:30:39.009599 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:39.009551    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-b45869bf8-wtj54" podStartSLOduration=2.009539612 podStartE2EDuration="2.009539612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:39.007874401 +0000 UTC m=+5878.576229188" watchObservedRunningTime="2026-11-03 00:30:39.009539612 +0000 UTC m=+5878.577894397"
Nov 03 00:30:39.864661 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:39.864625    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-rnnmk" podUID="cab95f3e-f51d-45b6-a218-e0c272a09aac" containerName="payload-processing" containerID="cri-o://eb3d73a0b4f614accf15ec9c40c2d1c88c796d2aeb56082a5e029dde8eddc3f0" gracePeriod=25
Nov 03 00:30:40.113658 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:40.113636    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-rnnmk"
Nov 03 00:30:40.202704 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:40.202634    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab95f3e-f51d-45b6-a218-e0c272a09aac-kube-api-access-sr42k\" (UniqueName: \"kubernetes.io/projected/cab95f3e-f51d-45b6-a218-e0c272a09aac-kube-api-access-sr42k\") pod \"cab95f3e-f51d-45b6-a218-e0c272a09aac\" (UID: \"cab95f3e-f51d-45b6-a218-e0c272a09aac\") "
Nov 03 00:30:40.202704 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:40.202680    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cab95f3e-f51d-45b6-a218-e0c272a09aac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab95f3e-f51d-45b6-a218-e0c272a09aac-plugins-config-volume\") pod \"cab95f3e-f51d-45b6-a218-e0c272a09aac\" (UID: \"cab95f3e-f51d-45b6-a218-e0c272a09aac\") "
Nov 03 00:30:40.203097 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:40.203068    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab95f3e-f51d-45b6-a218-e0c272a09aac-plugins-config-volume" pod "cab95f3e-f51d-45b6-a218-e0c272a09aac" (UID: "cab95f3e-f51d-45b6-a218-e0c272a09aac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:40.204593 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:40.204557    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab95f3e-f51d-45b6-a218-e0c272a09aac-kube-api-access-sr42k" pod "cab95f3e-f51d-45b6-a218-e0c272a09aac" (UID: "cab95f3e-f51d-45b6-a218-e0c272a09aac"). InnerVolumeSpecName "kube-api-access-sr42k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:40.303622 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:40.303591    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr42k\" (UniqueName: \"kubernetes.io/projected/cab95f3e-f51d-45b6-a218-e0c272a09aac-kube-api-access-sr42k\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:30:40.303622 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:40.303617    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab95f3e-f51d-45b6-a218-e0c272a09aac-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:30:40.790928 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:40.790898    4294 generic.go:356] "Generic (PLEG): container finished" podID="cab95f3e-f51d-45b6-a218-e0c272a09aac" containerID="eb3d73a0b4f614accf15ec9c40c2d1c88c796d2aeb56082a5e029dde8eddc3f0" exitCode=0
Nov 03 00:30:40.791123 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:40.790945    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-rnnmk"
Nov 03 00:30:40.791123 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:40.790996    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-rnnmk" event={"ID":"cab95f3e-f51d-45b6-a218-e0c272a09aac","Type":"ContainerDied","Data":"eb3d73a0b4f614accf15ec9c40c2d1c88c796d2aeb56082a5e029dde8eddc3f0"}
Nov 03 00:30:40.791123 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:40.791053    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-rnnmk" event={"ID":"cab95f3e-f51d-45b6-a218-e0c272a09aac","Type":"ContainerDied","Data":"3b78e03cb0c6b0a750b103bd8038e681b6106f6cf53342be92f55b9dc82fc7a9"}
Nov 03 00:30:40.791123 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:40.791082    4294 scope.go:122] "RemoveContainer" containerID="eb3d73a0b4f614accf15ec9c40c2d1c88c796d2aeb56082a5e029dde8eddc3f0"
Nov 03 00:30:40.799501 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:40.799486    4294 scope.go:122] "RemoveContainer" containerID="eb3d73a0b4f614accf15ec9c40c2d1c88c796d2aeb56082a5e029dde8eddc3f0"
Nov 03 00:30:40.799771 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:30:40.799752    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb3d73a0b4f614accf15ec9c40c2d1c88c796d2aeb56082a5e029dde8eddc3f0\": container with ID starting with eb3d73a0b4f614accf15ec9c40c2d1c88c796d2aeb56082a5e029dde8eddc3f0 not found: ID does not exist" containerID="eb3d73a0b4f614accf15ec9c40c2d1c88c796d2aeb56082a5e029dde8eddc3f0"
Nov 03 00:30:40.799850 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:40.799776    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb3d73a0b4f614accf15ec9c40c2d1c88c796d2aeb56082a5e029dde8eddc3f0"} err="failed to get container status \"eb3d73a0b4f614accf15ec9c40c2d1c88c796d2aeb56082a5e029dde8eddc3f0\": rpc error: code = NotFound desc = could not find container \"eb3d73a0b4f614accf15ec9c40c2d1c88c796d2aeb56082a5e029dde8eddc3f0\": container with ID starting with eb3d73a0b4f614accf15ec9c40c2d1c88c796d2aeb56082a5e029dde8eddc3f0 not found: ID does not exist"
Nov 03 00:30:40.948708 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:40.948634    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rnnmk"]
Nov 03 00:30:40.952852 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:40.952829    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rnnmk"]
Nov 03 00:30:41.748935 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:30:41.748898    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:30:42.546794 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:42.546763    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab95f3e-f51d-45b6-a218-e0c272a09aac" path="/var/lib/kubelet/pods/cab95f3e-f51d-45b6-a218-e0c272a09aac/volumes"
Nov 03 00:30:49.788147 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:49.788118    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-8499744ffb-z7wfd"
Nov 03 00:30:49.788560 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:49.788191    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-b45869bf8-wtj54"
Nov 03 00:30:49.949401 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:49.949372    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-78847578d5-wm8k4"]
Nov 03 00:30:50.036499 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:50.036454    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-59bd66d49-ddzr5"]
Nov 03 00:30:52.652749 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:52.652713    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:53.084417 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:53.084381    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:54.844429 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:54.844348    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:54.950623 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:54.950582    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-78847578d5-wm8k4" podUID="91945b75-548c-46b6-a76c-bde3e6afcb07" containerName="payload-processing" containerID="cri-o://bae9ec25ffeab0be3468a5c9d41925ad50c8961073bcdad1c98fea8e05eaad54" gracePeriod=25
Nov 03 00:30:55.037165 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.037102    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-59bd66d49-ddzr5" podUID="6f438f93-ef32-4dd5-9185-76f7a3a205ab" containerName="payload-pre-processing" containerID="cri-o://4e587996bd5e00d28299fe33e30c2384baed51bd52ea6c2f6bb4e80e6477919f" gracePeriod=25
Nov 03 00:30:55.287962 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.287937    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-78847578d5-wm8k4"
Nov 03 00:30:55.314569 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.314549    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59bd66d49-ddzr5"
Nov 03 00:30:55.345725 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.345704    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91945b75-548c-46b6-a76c-bde3e6afcb07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91945b75-548c-46b6-a76c-bde3e6afcb07-plugins-config-volume\") pod \"91945b75-548c-46b6-a76c-bde3e6afcb07\" (UID: \"91945b75-548c-46b6-a76c-bde3e6afcb07\") "
Nov 03 00:30:55.345876 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.345740    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91945b75-548c-46b6-a76c-bde3e6afcb07-kube-api-access-c5wrm\" (UniqueName: \"kubernetes.io/projected/91945b75-548c-46b6-a76c-bde3e6afcb07-kube-api-access-c5wrm\") pod \"91945b75-548c-46b6-a76c-bde3e6afcb07\" (UID: \"91945b75-548c-46b6-a76c-bde3e6afcb07\") "
Nov 03 00:30:55.346140 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.346117    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91945b75-548c-46b6-a76c-bde3e6afcb07-plugins-config-volume" pod "91945b75-548c-46b6-a76c-bde3e6afcb07" (UID: "91945b75-548c-46b6-a76c-bde3e6afcb07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:55.347795 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.347776    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91945b75-548c-46b6-a76c-bde3e6afcb07-kube-api-access-c5wrm" pod "91945b75-548c-46b6-a76c-bde3e6afcb07" (UID: "91945b75-548c-46b6-a76c-bde3e6afcb07"). InnerVolumeSpecName "kube-api-access-c5wrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:55.446634 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.446553    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f438f93-ef32-4dd5-9185-76f7a3a205ab-kube-api-access-c5t9n\" (UniqueName: \"kubernetes.io/projected/6f438f93-ef32-4dd5-9185-76f7a3a205ab-kube-api-access-c5t9n\") pod \"6f438f93-ef32-4dd5-9185-76f7a3a205ab\" (UID: \"6f438f93-ef32-4dd5-9185-76f7a3a205ab\") "
Nov 03 00:30:55.446810 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.446704    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f438f93-ef32-4dd5-9185-76f7a3a205ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f438f93-ef32-4dd5-9185-76f7a3a205ab-plugins-config-volume\") pod \"6f438f93-ef32-4dd5-9185-76f7a3a205ab\" (UID: \"6f438f93-ef32-4dd5-9185-76f7a3a205ab\") "
Nov 03 00:30:55.447014 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.446951    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91945b75-548c-46b6-a76c-bde3e6afcb07-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:30:55.447102 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.447022    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5wrm\" (UniqueName: \"kubernetes.io/projected/91945b75-548c-46b6-a76c-bde3e6afcb07-kube-api-access-c5wrm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:30:55.447144 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.447119    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f438f93-ef32-4dd5-9185-76f7a3a205ab-plugins-config-volume" pod "6f438f93-ef32-4dd5-9185-76f7a3a205ab" (UID: "6f438f93-ef32-4dd5-9185-76f7a3a205ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:55.449030 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.449007    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f438f93-ef32-4dd5-9185-76f7a3a205ab-kube-api-access-c5t9n" pod "6f438f93-ef32-4dd5-9185-76f7a3a205ab" (UID: "6f438f93-ef32-4dd5-9185-76f7a3a205ab"). InnerVolumeSpecName "kube-api-access-c5t9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:55.548132 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.548098    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f438f93-ef32-4dd5-9185-76f7a3a205ab-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:30:55.548132 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.548129    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5t9n\" (UniqueName: \"kubernetes.io/projected/6f438f93-ef32-4dd5-9185-76f7a3a205ab-kube-api-access-c5t9n\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:30:55.697819 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.697733    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:55.860890 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.860849    4294 generic.go:356] "Generic (PLEG): container finished" podID="91945b75-548c-46b6-a76c-bde3e6afcb07" containerID="bae9ec25ffeab0be3468a5c9d41925ad50c8961073bcdad1c98fea8e05eaad54" exitCode=0
Nov 03 00:30:55.861378 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.860896    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-78847578d5-wm8k4"
Nov 03 00:30:55.861378 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.860938    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-78847578d5-wm8k4" event={"ID":"91945b75-548c-46b6-a76c-bde3e6afcb07","Type":"ContainerDied","Data":"bae9ec25ffeab0be3468a5c9d41925ad50c8961073bcdad1c98fea8e05eaad54"}
Nov 03 00:30:55.861378 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.861008    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-78847578d5-wm8k4" event={"ID":"91945b75-548c-46b6-a76c-bde3e6afcb07","Type":"ContainerDied","Data":"e358d1ff37661bd60f1b814dc9d489546fdcb65f016f423104f623bcec9d4c1d"}
Nov 03 00:30:55.861378 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.861038    4294 scope.go:122] "RemoveContainer" containerID="bae9ec25ffeab0be3468a5c9d41925ad50c8961073bcdad1c98fea8e05eaad54"
Nov 03 00:30:55.862537 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.862517    4294 generic.go:356] "Generic (PLEG): container finished" podID="6f438f93-ef32-4dd5-9185-76f7a3a205ab" containerID="4e587996bd5e00d28299fe33e30c2384baed51bd52ea6c2f6bb4e80e6477919f" exitCode=0
Nov 03 00:30:55.862665 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.862563    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59bd66d49-ddzr5" event={"ID":"6f438f93-ef32-4dd5-9185-76f7a3a205ab","Type":"ContainerDied","Data":"4e587996bd5e00d28299fe33e30c2384baed51bd52ea6c2f6bb4e80e6477919f"}
Nov 03 00:30:55.862665 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.862574    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59bd66d49-ddzr5"
Nov 03 00:30:55.862665 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.862596    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59bd66d49-ddzr5" event={"ID":"6f438f93-ef32-4dd5-9185-76f7a3a205ab","Type":"ContainerDied","Data":"163a4362307e04372a0e23c8405ab7009bab5dcfbb8b2cf148a471b97cb0028b"}
Nov 03 00:30:55.872317 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.872299    4294 scope.go:122] "RemoveContainer" containerID="bae9ec25ffeab0be3468a5c9d41925ad50c8961073bcdad1c98fea8e05eaad54"
Nov 03 00:30:55.872606 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:30:55.872578    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae9ec25ffeab0be3468a5c9d41925ad50c8961073bcdad1c98fea8e05eaad54\": container with ID starting with bae9ec25ffeab0be3468a5c9d41925ad50c8961073bcdad1c98fea8e05eaad54 not found: ID does not exist" containerID="bae9ec25ffeab0be3468a5c9d41925ad50c8961073bcdad1c98fea8e05eaad54"
Nov 03 00:30:55.872715 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.872616    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae9ec25ffeab0be3468a5c9d41925ad50c8961073bcdad1c98fea8e05eaad54"} err="failed to get container status \"bae9ec25ffeab0be3468a5c9d41925ad50c8961073bcdad1c98fea8e05eaad54\": rpc error: code = NotFound desc = could not find container \"bae9ec25ffeab0be3468a5c9d41925ad50c8961073bcdad1c98fea8e05eaad54\": container with ID starting with bae9ec25ffeab0be3468a5c9d41925ad50c8961073bcdad1c98fea8e05eaad54 not found: ID does not exist"
Nov 03 00:30:55.872715 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.872641    4294 scope.go:122] "RemoveContainer" containerID="4e587996bd5e00d28299fe33e30c2384baed51bd52ea6c2f6bb4e80e6477919f"
Nov 03 00:30:55.880702 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.880684    4294 scope.go:122] "RemoveContainer" containerID="4e587996bd5e00d28299fe33e30c2384baed51bd52ea6c2f6bb4e80e6477919f"
Nov 03 00:30:55.880964 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:30:55.880946    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e587996bd5e00d28299fe33e30c2384baed51bd52ea6c2f6bb4e80e6477919f\": container with ID starting with 4e587996bd5e00d28299fe33e30c2384baed51bd52ea6c2f6bb4e80e6477919f not found: ID does not exist" containerID="4e587996bd5e00d28299fe33e30c2384baed51bd52ea6c2f6bb4e80e6477919f"
Nov 03 00:30:55.881026 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:55.880972    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e587996bd5e00d28299fe33e30c2384baed51bd52ea6c2f6bb4e80e6477919f"} err="failed to get container status \"4e587996bd5e00d28299fe33e30c2384baed51bd52ea6c2f6bb4e80e6477919f\": rpc error: code = NotFound desc = could not find container \"4e587996bd5e00d28299fe33e30c2384baed51bd52ea6c2f6bb4e80e6477919f\": container with ID starting with 4e587996bd5e00d28299fe33e30c2384baed51bd52ea6c2f6bb4e80e6477919f not found: ID does not exist"
Nov 03 00:30:56.018646 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.018612    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-59bd66d49-ddzr5"]
Nov 03 00:30:56.022668 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.022643    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-59bd66d49-ddzr5"]
Nov 03 00:30:56.126599 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.126565    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-78847578d5-wm8k4"]
Nov 03 00:30:56.131587 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.131558    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-78847578d5-wm8k4"]
Nov 03 00:30:56.156864 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.156834    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"]
Nov 03 00:30:56.157226 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.157214    4294 cpu_manager.go:414] "containerMap: removing container" podUID="cab95f3e-f51d-45b6-a218-e0c272a09aac" containerName="payload-processing"
Nov 03 00:30:56.157285 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.157228    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab95f3e-f51d-45b6-a218-e0c272a09aac" containerName="payload-processing"
Nov 03 00:30:56.157285 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.157240    4294 cpu_manager.go:414] "containerMap: removing container" podUID="91945b75-548c-46b6-a76c-bde3e6afcb07" containerName="payload-processing"
Nov 03 00:30:56.157285 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.157247    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91945b75-548c-46b6-a76c-bde3e6afcb07" containerName="payload-processing"
Nov 03 00:30:56.157285 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.157265    4294 cpu_manager.go:414] "containerMap: removing container" podUID="6f438f93-ef32-4dd5-9185-76f7a3a205ab" containerName="payload-pre-processing"
Nov 03 00:30:56.157285 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.157271    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f438f93-ef32-4dd5-9185-76f7a3a205ab" containerName="payload-pre-processing"
Nov 03 00:30:56.157444 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.157323    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f438f93-ef32-4dd5-9185-76f7a3a205ab" containerName="payload-pre-processing"
Nov 03 00:30:56.157444 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.157335    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab95f3e-f51d-45b6-a218-e0c272a09aac" containerName="payload-processing"
Nov 03 00:30:56.157444 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.157341    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="91945b75-548c-46b6-a76c-bde3e6afcb07" containerName="payload-processing"
Nov 03 00:30:56.161666 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.161649    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"
Nov 03 00:30:56.213025 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.212972    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\""
Nov 03 00:30:56.213218 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.213136    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 00:30:56.213218 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.213212    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 00:30:56.213331 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.213241    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 00:30:56.215326 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.215279    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"]
Nov 03 00:30:56.225957 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.225936    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 00:30:56.226085 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.226071    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-v5z29\""
Nov 03 00:30:56.254551 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.254522    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbd113d1-32fd-49f4-87df-a1c8e8541aff-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-ghbc7\" (UID: \"fbd113d1-32fd-49f4-87df-a1c8e8541aff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"
Nov 03 00:30:56.254674 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.254576    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd113d1-32fd-49f4-87df-a1c8e8541aff-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-ghbc7\" (UID: \"fbd113d1-32fd-49f4-87df-a1c8e8541aff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"
Nov 03 00:30:56.254674 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.254661    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbd113d1-32fd-49f4-87df-a1c8e8541aff-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-ghbc7\" (UID: \"fbd113d1-32fd-49f4-87df-a1c8e8541aff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"
Nov 03 00:30:56.254743 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.254684    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cvxh\" (UniqueName: \"kubernetes.io/projected/fbd113d1-32fd-49f4-87df-a1c8e8541aff-kube-api-access-9cvxh\") pod \"payload-processing-external-model-5bcd7997-ghbc7\" (UID: \"fbd113d1-32fd-49f4-87df-a1c8e8541aff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"
Nov 03 00:30:56.355753 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.355714    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cvxh\" (UniqueName: \"kubernetes.io/projected/fbd113d1-32fd-49f4-87df-a1c8e8541aff-kube-api-access-9cvxh\") pod \"payload-processing-external-model-5bcd7997-ghbc7\" (UID: \"fbd113d1-32fd-49f4-87df-a1c8e8541aff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"
Nov 03 00:30:56.355934 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.355842    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbd113d1-32fd-49f4-87df-a1c8e8541aff-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-ghbc7\" (UID: \"fbd113d1-32fd-49f4-87df-a1c8e8541aff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"
Nov 03 00:30:56.355934 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.355897    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd113d1-32fd-49f4-87df-a1c8e8541aff-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-ghbc7\" (UID: \"fbd113d1-32fd-49f4-87df-a1c8e8541aff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"
Nov 03 00:30:56.356052 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.355936    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbd113d1-32fd-49f4-87df-a1c8e8541aff-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-ghbc7\" (UID: \"fbd113d1-32fd-49f4-87df-a1c8e8541aff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"
Nov 03 00:30:56.356586 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.356554    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbd113d1-32fd-49f4-87df-a1c8e8541aff-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-ghbc7\" (UID: \"fbd113d1-32fd-49f4-87df-a1c8e8541aff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"
Nov 03 00:30:56.356718 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.356701    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd113d1-32fd-49f4-87df-a1c8e8541aff-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-ghbc7\" (UID: \"fbd113d1-32fd-49f4-87df-a1c8e8541aff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"
Nov 03 00:30:56.358318 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.358301    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbd113d1-32fd-49f4-87df-a1c8e8541aff-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-ghbc7\" (UID: \"fbd113d1-32fd-49f4-87df-a1c8e8541aff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"
Nov 03 00:30:56.410851 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.410824    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cvxh\" (UniqueName: \"kubernetes.io/projected/fbd113d1-32fd-49f4-87df-a1c8e8541aff-kube-api-access-9cvxh\") pod \"payload-processing-external-model-5bcd7997-ghbc7\" (UID: \"fbd113d1-32fd-49f4-87df-a1c8e8541aff\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"
Nov 03 00:30:56.474158 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.474077    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"
Nov 03 00:30:56.574433 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.568619    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f438f93-ef32-4dd5-9185-76f7a3a205ab" path="/var/lib/kubelet/pods/6f438f93-ef32-4dd5-9185-76f7a3a205ab/volumes"
Nov 03 00:30:56.574433 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.569471    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91945b75-548c-46b6-a76c-bde3e6afcb07" path="/var/lib/kubelet/pods/91945b75-548c-46b6-a76c-bde3e6afcb07/volumes"
Nov 03 00:30:56.851671 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.851642    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"]
Nov 03 00:30:56.853021 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:30:56.852996    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd113d1_32fd_49f4_87df_a1c8e8541aff.slice/crio-c098aa7d555295a78139154592542fcd5128b6c9be4bb1b042971b02e21f663d WatchSource:0}: Error finding container c098aa7d555295a78139154592542fcd5128b6c9be4bb1b042971b02e21f663d: Status 404 returned error can't find the container with id c098aa7d555295a78139154592542fcd5128b6c9be4bb1b042971b02e21f663d
Nov 03 00:30:56.868697 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:56.868663    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7" event={"ID":"fbd113d1-32fd-49f4-87df-a1c8e8541aff","Type":"ContainerStarted","Data":"c098aa7d555295a78139154592542fcd5128b6c9be4bb1b042971b02e21f663d"}
Nov 03 00:30:57.877301 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:57.877263    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7" event={"ID":"fbd113d1-32fd-49f4-87df-a1c8e8541aff","Type":"ContainerStarted","Data":"69dd617f1306abb4d40e01e530484b86f2ce13e7897f2bc4a3850b58e0f9a7d9"}
Nov 03 00:30:57.877716 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:57.877426    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"
Nov 03 00:30:58.043493 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:30:58.043440    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7" podStartSLOduration=2.043425097 podStartE2EDuration="2.043425097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:58.040299462 +0000 UTC m=+5897.608654249" watchObservedRunningTime="2026-11-03 00:30:58.043425097 +0000 UTC m=+5897.611779884"
Nov 03 00:31:08.886134 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:31:08.886102    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"
Nov 03 00:31:11.751639 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:31:11.751548    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:31:40.924944 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:31:40.924908    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:41.748512 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:31:41.748476    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:31:44.731337 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:31:44.731308    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:55.143747 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:31:55.143712    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:11.748440 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:32:11.748405    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:32:41.748707 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:32:41.748664    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:32:49.604173 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:49.604139    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"]
Nov 03 00:32:49.789093 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:49.789059    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:49.962909 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:49.962828    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:50.313759 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:50.313730    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:54.607103 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:54.607049    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7" podUID="fbd113d1-32fd-49f4-87df-a1c8e8541aff" containerName="payload-processing" containerID="cri-o://69dd617f1306abb4d40e01e530484b86f2ce13e7897f2bc4a3850b58e0f9a7d9" gracePeriod=25
Nov 03 00:32:55.064611 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.064291    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"
Nov 03 00:32:55.205466 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.205385    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd113d1-32fd-49f4-87df-a1c8e8541aff-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbd113d1-32fd-49f4-87df-a1c8e8541aff-provider-credentials\") pod \"fbd113d1-32fd-49f4-87df-a1c8e8541aff\" (UID: \"fbd113d1-32fd-49f4-87df-a1c8e8541aff\") "
Nov 03 00:32:55.205466 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.205431    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd113d1-32fd-49f4-87df-a1c8e8541aff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd113d1-32fd-49f4-87df-a1c8e8541aff-plugins-config-volume\") pod \"fbd113d1-32fd-49f4-87df-a1c8e8541aff\" (UID: \"fbd113d1-32fd-49f4-87df-a1c8e8541aff\") "
Nov 03 00:32:55.205466 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.205463    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd113d1-32fd-49f4-87df-a1c8e8541aff-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbd113d1-32fd-49f4-87df-a1c8e8541aff-routing-overlay-volume\") pod \"fbd113d1-32fd-49f4-87df-a1c8e8541aff\" (UID: \"fbd113d1-32fd-49f4-87df-a1c8e8541aff\") "
Nov 03 00:32:55.205750 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.205551    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd113d1-32fd-49f4-87df-a1c8e8541aff-kube-api-access-9cvxh\" (UniqueName: \"kubernetes.io/projected/fbd113d1-32fd-49f4-87df-a1c8e8541aff-kube-api-access-9cvxh\") pod \"fbd113d1-32fd-49f4-87df-a1c8e8541aff\" (UID: \"fbd113d1-32fd-49f4-87df-a1c8e8541aff\") "
Nov 03 00:32:55.205869 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.205842    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd113d1-32fd-49f4-87df-a1c8e8541aff-plugins-config-volume" pod "fbd113d1-32fd-49f4-87df-a1c8e8541aff" (UID: "fbd113d1-32fd-49f4-87df-a1c8e8541aff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:55.205937 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.205910    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd113d1-32fd-49f4-87df-a1c8e8541aff-routing-overlay-volume" pod "fbd113d1-32fd-49f4-87df-a1c8e8541aff" (UID: "fbd113d1-32fd-49f4-87df-a1c8e8541aff"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:55.207778 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.207747    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd113d1-32fd-49f4-87df-a1c8e8541aff-provider-credentials" pod "fbd113d1-32fd-49f4-87df-a1c8e8541aff" (UID: "fbd113d1-32fd-49f4-87df-a1c8e8541aff"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:55.207893 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.207801    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd113d1-32fd-49f4-87df-a1c8e8541aff-kube-api-access-9cvxh" pod "fbd113d1-32fd-49f4-87df-a1c8e8541aff" (UID: "fbd113d1-32fd-49f4-87df-a1c8e8541aff"). InnerVolumeSpecName "kube-api-access-9cvxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:55.306511 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.306487    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cvxh\" (UniqueName: \"kubernetes.io/projected/fbd113d1-32fd-49f4-87df-a1c8e8541aff-kube-api-access-9cvxh\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:32:55.306511 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.306508    4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbd113d1-32fd-49f4-87df-a1c8e8541aff-provider-credentials\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:32:55.306653 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.306518    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd113d1-32fd-49f4-87df-a1c8e8541aff-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:32:55.306653 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.306529    4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbd113d1-32fd-49f4-87df-a1c8e8541aff-routing-overlay-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:32:55.512363 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.512328    4294 generic.go:356] "Generic (PLEG): container finished" podID="fbd113d1-32fd-49f4-87df-a1c8e8541aff" containerID="69dd617f1306abb4d40e01e530484b86f2ce13e7897f2bc4a3850b58e0f9a7d9" exitCode=0
Nov 03 00:32:55.512524 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.512372    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"
Nov 03 00:32:55.512524 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.512404    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7" event={"ID":"fbd113d1-32fd-49f4-87df-a1c8e8541aff","Type":"ContainerDied","Data":"69dd617f1306abb4d40e01e530484b86f2ce13e7897f2bc4a3850b58e0f9a7d9"}
Nov 03 00:32:55.512524 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.512448    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7" event={"ID":"fbd113d1-32fd-49f4-87df-a1c8e8541aff","Type":"ContainerDied","Data":"c098aa7d555295a78139154592542fcd5128b6c9be4bb1b042971b02e21f663d"}
Nov 03 00:32:55.512524 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.512474    4294 scope.go:122] "RemoveContainer" containerID="69dd617f1306abb4d40e01e530484b86f2ce13e7897f2bc4a3850b58e0f9a7d9"
Nov 03 00:32:55.523937 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.523843    4294 scope.go:122] "RemoveContainer" containerID="69dd617f1306abb4d40e01e530484b86f2ce13e7897f2bc4a3850b58e0f9a7d9"
Nov 03 00:32:55.524159 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:32:55.524128    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69dd617f1306abb4d40e01e530484b86f2ce13e7897f2bc4a3850b58e0f9a7d9\": container with ID starting with 69dd617f1306abb4d40e01e530484b86f2ce13e7897f2bc4a3850b58e0f9a7d9 not found: ID does not exist" containerID="69dd617f1306abb4d40e01e530484b86f2ce13e7897f2bc4a3850b58e0f9a7d9"
Nov 03 00:32:55.524286 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.524170    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69dd617f1306abb4d40e01e530484b86f2ce13e7897f2bc4a3850b58e0f9a7d9"} err="failed to get container status \"69dd617f1306abb4d40e01e530484b86f2ce13e7897f2bc4a3850b58e0f9a7d9\": rpc error: code = NotFound desc = could not find container \"69dd617f1306abb4d40e01e530484b86f2ce13e7897f2bc4a3850b58e0f9a7d9\": container with ID starting with 69dd617f1306abb4d40e01e530484b86f2ce13e7897f2bc4a3850b58e0f9a7d9 not found: ID does not exist"
Nov 03 00:32:55.606904 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.606879    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:55.669748 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.669720    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"]
Nov 03 00:32:55.672738 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:55.672714    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-ghbc7"]
Nov 03 00:32:56.545668 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:32:56.545637    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd113d1-32fd-49f4-87df-a1c8e8541aff" path="/var/lib/kubelet/pods/fbd113d1-32fd-49f4-87df-a1c8e8541aff/volumes"
Nov 03 00:33:11.747690 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:33:11.747653    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:33:22.245062 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:33:22.245024    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:41.750311 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:33:41.750272    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:33:55.920513 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:33:55.920420    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:11.749554 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:34:11.749516    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:34:18.615865 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:34:18.615829    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:41.751823 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:34:41.751779    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:34:56.249843 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:34:56.249805    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:56.403304 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:34:56.403264    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:11.752193 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:35:11.752142    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:35:41.749288 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:35:41.749249    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:35:43.443383 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:35:43.443344    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:56.603799 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:35:56.603766    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:11.749488 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:36:11.749449    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:36:41.752993 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:36:41.752944    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:36:56.915144 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:36:56.915055    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:11.749106 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:37:11.749067    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:37:41.749551 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:37:41.749513    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:37:57.234855 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:37:57.234811    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:57.400603 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:37:57.400571    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:01.735392 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:01.735338    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:02.191846 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.191808    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"]
Nov 03 00:38:02.192412 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.192395    4294 cpu_manager.go:414] "containerMap: removing container" podUID="fbd113d1-32fd-49f4-87df-a1c8e8541aff" containerName="payload-processing"
Nov 03 00:38:02.192467 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.192414    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd113d1-32fd-49f4-87df-a1c8e8541aff" containerName="payload-processing"
Nov 03 00:38:02.192509 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.192498    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd113d1-32fd-49f4-87df-a1c8e8541aff" containerName="payload-processing"
Nov 03 00:38:02.195653 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.195637    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.245657 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.245621    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-aa4c43-openshift-default-dockercfg-wkhfx\""
Nov 03 00:38:02.295529 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.295492    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"]
Nov 03 00:38:02.313735 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.313705    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:02.349489 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.349442    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-workload-socket\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.349714 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.349572    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aebb0a8-32c9-409a-beaa-721ce423294b-istiod-ca-cert\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.349714 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.349649    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-envoy\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.349714 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.349691    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-podinfo\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.349879 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.349722    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6s4n\" (UniqueName: \"kubernetes.io/projected/2aebb0a8-32c9-409a-beaa-721ce423294b-kube-api-access-x6s4n\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.349879 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.349769    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-data\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.349879 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.349858    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-token\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.350081 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.349916    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-workload-certs\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.350081 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.349945    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-credential-socket\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.451439 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.451338    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-token\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.451439 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.451404    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-workload-certs\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.451681 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.451517    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-credential-socket\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.451681 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.451609    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-workload-socket\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.451681 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.451661    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aebb0a8-32c9-409a-beaa-721ce423294b-istiod-ca-cert\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.451876 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.451699    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-envoy\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.451876 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.451734    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-podinfo\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.451876 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.451764    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6s4n\" (UniqueName: \"kubernetes.io/projected/2aebb0a8-32c9-409a-beaa-721ce423294b-kube-api-access-x6s4n\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.451876 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.451817    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-data\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.451876 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.451827    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-workload-certs\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.452137 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.451905    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-credential-socket\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.452137 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.451974    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-workload-socket\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.452241 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.452201    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-data\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.452329 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.452310    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aebb0a8-32c9-409a-beaa-721ce423294b-istiod-ca-cert\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.454169 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.454146    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-envoy\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.454390 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.454371    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-podinfo\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.511225 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.511188    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-token\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.511392 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.511344    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6s4n\" (UniqueName: \"kubernetes.io/projected/2aebb0a8-32c9-409a-beaa-721ce423294b-kube-api-access-x6s4n\") pod \"e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") " pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:02.606632 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.606602    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:02.808584 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:02.808552    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:03.220876 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:03.220848    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"]
Nov 03 00:38:03.221663 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:38:03.221633    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aebb0a8_32c9_409a_beaa_721ce423294b.slice/crio-0f9ba59d961b09e0bd92944aab25589dafa9a300c1eb88c03bceadb3f1ed0065 WatchSource:0}: Error finding container 0f9ba59d961b09e0bd92944aab25589dafa9a300c1eb88c03bceadb3f1ed0065: Status 404 returned error can't find the container with id 0f9ba59d961b09e0bd92944aab25589dafa9a300c1eb88c03bceadb3f1ed0065
Nov 03 00:38:03.223822 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:03.223792    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:38:03.223883 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:03.223857    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:38:03.223924 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:03.223887    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:38:04.204412 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:04.204363    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx" event={"ID":"2aebb0a8-32c9-409a-beaa-721ce423294b","Type":"ContainerStarted","Data":"2742dffae6baa24f5b7efab77dd1e9856ec2e8eb39a2f20ad960fdcfb6d5bb32"}
Nov 03 00:38:04.204835 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:04.204427    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx" event={"ID":"2aebb0a8-32c9-409a-beaa-721ce423294b","Type":"ContainerStarted","Data":"0f9ba59d961b09e0bd92944aab25589dafa9a300c1eb88c03bceadb3f1ed0065"}
Nov 03 00:38:04.365131 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:04.365073    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx" podStartSLOduration=2.36505885 podStartE2EDuration="2.36505885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:04.36402789 +0000 UTC m=+6323.932382677" watchObservedRunningTime="2026-11-03 00:38:04.36505885 +0000 UTC m=+6323.933413636"
Nov 03 00:38:04.809166 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:04.809122    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:04.814776 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:04.814747    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:05.209643 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:05.209564    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:05.210560 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:05.210544    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:07.340774 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:07.340738    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:07.514856 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:07.514820    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:07.697095 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:07.696959    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:10.435011 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:10.434960    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:10.596684 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:10.596652    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:10.785188 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:10.785134    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:10.934744 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:10.934712    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb"]
Nov 03 00:38:10.938424 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:10.938404    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb"
Nov 03 00:38:10.993967 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:10.993932    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb"]
Nov 03 00:38:11.036443 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.036348    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-aa4c43\""
Nov 03 00:38:11.036443 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.036421    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-3d5ef5e7-dockercfg-hph8p\""
Nov 03 00:38:11.036645 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.036609    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3bd798-95c8-46bc-8189-0a65252d2911-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb\" (UID: \"dc3bd798-95c8-46bc-8189-0a65252d2911\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb"
Nov 03 00:38:11.036645 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.036637    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blw26\" (UniqueName: \"kubernetes.io/projected/dc3bd798-95c8-46bc-8189-0a65252d2911-kube-api-access-blw26\") pod \"payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb\" (UID: \"dc3bd798-95c8-46bc-8189-0a65252d2911\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb"
Nov 03 00:38:11.098782 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.098739    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88"]
Nov 03 00:38:11.102378 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.102353    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88"
Nov 03 00:38:11.137441 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.137406    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3bd798-95c8-46bc-8189-0a65252d2911-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb\" (UID: \"dc3bd798-95c8-46bc-8189-0a65252d2911\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb"
Nov 03 00:38:11.137441 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.137442    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blw26\" (UniqueName: \"kubernetes.io/projected/dc3bd798-95c8-46bc-8189-0a65252d2911-kube-api-access-blw26\") pod \"payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb\" (UID: \"dc3bd798-95c8-46bc-8189-0a65252d2911\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb"
Nov 03 00:38:11.137671 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.137504    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88\" (UID: \"cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88"
Nov 03 00:38:11.137671 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.137603    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7nqc\" (UniqueName: \"kubernetes.io/projected/cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6-kube-api-access-w7nqc\") pod \"payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88\" (UID: \"cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88"
Nov 03 00:38:11.138149 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.138131    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3bd798-95c8-46bc-8189-0a65252d2911-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb\" (UID: \"dc3bd798-95c8-46bc-8189-0a65252d2911\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb"
Nov 03 00:38:11.156127 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.156100    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88"]
Nov 03 00:38:11.194693 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.194665    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blw26\" (UniqueName: \"kubernetes.io/projected/dc3bd798-95c8-46bc-8189-0a65252d2911-kube-api-access-blw26\") pod \"payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb\" (UID: \"dc3bd798-95c8-46bc-8189-0a65252d2911\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb"
Nov 03 00:38:11.238276 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.238248    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7nqc\" (UniqueName: \"kubernetes.io/projected/cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6-kube-api-access-w7nqc\") pod \"payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88\" (UID: \"cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88"
Nov 03 00:38:11.238464 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.238340    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88\" (UID: \"cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88"
Nov 03 00:38:11.239146 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.239121    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88\" (UID: \"cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88"
Nov 03 00:38:11.251726 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.251702    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb"
Nov 03 00:38:11.293221 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.293147    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7nqc\" (UniqueName: \"kubernetes.io/projected/cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6-kube-api-access-w7nqc\") pod \"payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88\" (UID: \"cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88"
Nov 03 00:38:11.415422 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.415389    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88"
Nov 03 00:38:11.618623 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.618600    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb"]
Nov 03 00:38:11.619873 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:38:11.619851    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3bd798_95c8_46bc_8189_0a65252d2911.slice/crio-12fa6d8268fbc1f850cb9233c845648032544d4116ddcc3ef06be3fa8536fb5d WatchSource:0}: Error finding container 12fa6d8268fbc1f850cb9233c845648032544d4116ddcc3ef06be3fa8536fb5d: Status 404 returned error can't find the container with id 12fa6d8268fbc1f850cb9233c845648032544d4116ddcc3ef06be3fa8536fb5d
Nov 03 00:38:11.749380 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:38:11.749345    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:38:11.778132 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:11.778107    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88"]
Nov 03 00:38:11.779022 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:38:11.778996    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf767a8_4020_4f8b_b2c1_4abb5ab90aa6.slice/crio-611a236beeb8ff778901093e849b3d35e0a183131792dc22f3ed3fae2a0b41aa WatchSource:0}: Error finding container 611a236beeb8ff778901093e849b3d35e0a183131792dc22f3ed3fae2a0b41aa: Status 404 returned error can't find the container with id 611a236beeb8ff778901093e849b3d35e0a183131792dc22f3ed3fae2a0b41aa
Nov 03 00:38:12.248844 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:12.248805    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88" event={"ID":"cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6","Type":"ContainerStarted","Data":"01923b28ea5359cc0f43f2adf86bc477315f567bb36ca24b8a31995cda58641e"}
Nov 03 00:38:12.249082 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:12.248869    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88" event={"ID":"cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6","Type":"ContainerStarted","Data":"611a236beeb8ff778901093e849b3d35e0a183131792dc22f3ed3fae2a0b41aa"}
Nov 03 00:38:12.249082 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:12.248902    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88"
Nov 03 00:38:12.250437 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:12.250412    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb" event={"ID":"dc3bd798-95c8-46bc-8189-0a65252d2911","Type":"ContainerStarted","Data":"dbfd29df2f10dc1b1f03d24e0e9d7266d475b959933099cb59d5f2727cfe1f2a"}
Nov 03 00:38:12.250553 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:12.250452    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb" event={"ID":"dc3bd798-95c8-46bc-8189-0a65252d2911","Type":"ContainerStarted","Data":"12fa6d8268fbc1f850cb9233c845648032544d4116ddcc3ef06be3fa8536fb5d"}
Nov 03 00:38:12.250553 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:12.250499    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb"
Nov 03 00:38:12.364393 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:12.364336    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88" podStartSLOduration=1.364315807 podStartE2EDuration="1.364315807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:12.362053148 +0000 UTC m=+6331.930407935" watchObservedRunningTime="2026-11-03 00:38:12.364315807 +0000 UTC m=+6331.932670598"
Nov 03 00:38:12.473287 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:12.473230    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb" podStartSLOduration=2.473211172 podStartE2EDuration="2.473211172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:12.471637337 +0000 UTC m=+6332.039992125" watchObservedRunningTime="2026-11-03 00:38:12.473211172 +0000 UTC m=+6332.041565995"
Nov 03 00:38:15.576793 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:15.576761    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:20.578005 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:20.574570    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:23.278963 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:23.278935    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb"
Nov 03 00:38:23.279442 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:23.279097    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88"
Nov 03 00:38:25.570177 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:25.570134    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:28.329635 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:28.329597    4294 generic.go:356] "Generic (PLEG): container finished" podID="2aebb0a8-32c9-409a-beaa-721ce423294b" containerID="2742dffae6baa24f5b7efab77dd1e9856ec2e8eb39a2f20ad960fdcfb6d5bb32" exitCode=0
Nov 03 00:38:28.330218 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:28.329643    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx" event={"ID":"2aebb0a8-32c9-409a-beaa-721ce423294b","Type":"ContainerDied","Data":"2742dffae6baa24f5b7efab77dd1e9856ec2e8eb39a2f20ad960fdcfb6d5bb32"}
Nov 03 00:38:28.330218 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:28.330026    4294 scope.go:122] "RemoveContainer" containerID="2742dffae6baa24f5b7efab77dd1e9856ec2e8eb39a2f20ad960fdcfb6d5bb32"
Nov 03 00:38:28.331288 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:28.331258    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:38:28.331359 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:28.331330    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:38:28.331403 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:28.331368    4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:38:28.978019 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:28.976128    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:29.336571 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:29.336535    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx" event={"ID":"2aebb0a8-32c9-409a-beaa-721ce423294b","Type":"ContainerStarted","Data":"6b7c0e159140021b767563b60f7c727c424577313d7745d1fcecf118d8c44c99"}
Nov 03 00:38:29.809144 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:29.809111    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:29.809339 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:29.809305    4294 patch_prober.go:29] interesting pod/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 00:38:29.809394 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:29.809347    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx" podUID="2aebb0a8-32c9-409a-beaa-721ce423294b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 00:38:30.810007 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:30.809940    4294 patch_prober.go:29] interesting pod/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 00:38:30.810444 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:30.810028    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx" podUID="2aebb0a8-32c9-409a-beaa-721ce423294b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 00:38:30.980808 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:30.980767    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"]
Nov 03 00:38:30.983520 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:30.983495    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:30.989097 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:30.989067    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:30.992670 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:30.992580    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a1aea84-5478-487a-9635-fab8b9a81696-tls-certs\") pod \"simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") " pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:30.992670 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:30.992638    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-tmp-dir\") pod \"simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") " pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:30.992670 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:30.992664    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-home\") pod \"simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") " pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:30.992890 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:30.992679    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-model-cache\") pod \"simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") " pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:30.992890 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:30.992735    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-dshm\") pod \"simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") " pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:30.992890 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:30.992815    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tcl7\" (UniqueName: \"kubernetes.io/projected/4a1aea84-5478-487a-9635-fab8b9a81696-kube-api-access-4tcl7\") pod \"simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") " pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:31.082551 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.082460    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-aa4c43\"/\"kube-root-ca.crt\""
Nov 03 00:38:31.082715 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.082576    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-aa4c43\"/\"simulator-single-main-aa4c43-kserve-self-signed-certs\""
Nov 03 00:38:31.082715 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.082578    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-aa4c43\"/\"openshift-service-ca.crt\""
Nov 03 00:38:31.082994 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.082957    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-aa4c43\"/\"default-dockercfg-nd7q5\""
Nov 03 00:38:31.085040 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.085023    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"]
Nov 03 00:38:31.093415 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.093390    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-dshm\") pod \"simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") " pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:31.093541 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.093463    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tcl7\" (UniqueName: \"kubernetes.io/projected/4a1aea84-5478-487a-9635-fab8b9a81696-kube-api-access-4tcl7\") pod \"simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") " pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:31.093541 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.093484    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a1aea84-5478-487a-9635-fab8b9a81696-tls-certs\") pod \"simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") " pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:31.093541 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.093518    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-tmp-dir\") pod \"simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") " pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:31.093541 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.093537    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-home\") pod \"simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") " pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:31.093758 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.093558    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-model-cache\") pod \"simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") " pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:31.093918 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.093896    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-tmp-dir\") pod \"simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") " pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:31.094007 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.093926    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-home\") pod \"simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") " pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:31.094007 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.093971    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-model-cache\") pod \"simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") " pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:31.095847 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.095829    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-dshm\") pod \"simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") " pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:31.096141 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.096120    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a1aea84-5478-487a-9635-fab8b9a81696-tls-certs\") pod \"simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") " pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:31.147934 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.147903    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tcl7\" (UniqueName: \"kubernetes.io/projected/4a1aea84-5478-487a-9635-fab8b9a81696-kube-api-access-4tcl7\") pod \"simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") " pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:31.152736 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.152709    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:31.297699 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.297668    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:31.678161 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.678067    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"]
Nov 03 00:38:31.681839 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:38:31.681806    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a1aea84_5478_487a_9635_fab8b9a81696.slice/crio-7ccf98529571e7c33d56f9e4378598bd4fec0e3bff31a96f18a387ca5677fcaf WatchSource:0}: Error finding container 7ccf98529571e7c33d56f9e4378598bd4fec0e3bff31a96f18a387ca5677fcaf: Status 404 returned error can't find the container with id 7ccf98529571e7c33d56f9e4378598bd4fec0e3bff31a96f18a387ca5677fcaf
Nov 03 00:38:31.809029 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.808989    4294 patch_prober.go:29] interesting pod/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 00:38:31.809217 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:31.809048    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx" podUID="2aebb0a8-32c9-409a-beaa-721ce423294b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 00:38:32.378309 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:32.378268    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd" event={"ID":"4a1aea84-5478-487a-9635-fab8b9a81696","Type":"ContainerStarted","Data":"9db8df7ef30546701653123329a760867f261033a8e57758a7416e6b8f7941e1"}
Nov 03 00:38:32.378309 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:32.378323    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd" event={"ID":"4a1aea84-5478-487a-9635-fab8b9a81696","Type":"ContainerStarted","Data":"7ccf98529571e7c33d56f9e4378598bd4fec0e3bff31a96f18a387ca5677fcaf"}
Nov 03 00:38:32.492329 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:32.492269    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd" podStartSLOduration=2.492253736 podStartE2EDuration="2.492253736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:32.490789547 +0000 UTC m=+6352.059144339" watchObservedRunningTime="2026-11-03 00:38:32.492253736 +0000 UTC m=+6352.060608523"
Nov 03 00:38:32.809715 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:32.809674    4294 patch_prober.go:29] interesting pod/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 00:38:32.809925 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:32.809746    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx" podUID="2aebb0a8-32c9-409a-beaa-721ce423294b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 00:38:33.297817 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:33.297776    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:33.809489 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:33.809443    4294 patch_prober.go:29] interesting pod/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 00:38:33.809896 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:33.809512    4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx" podUID="2aebb0a8-32c9-409a-beaa-721ce423294b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 00:38:34.557429 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:34.557396    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:34.813708 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:34.813629    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:34.814226 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:34.813909    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:34.814600 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:34.814583    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:38:35.897187 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:35.897152    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:40.876436 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:40.876400    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:41.298093 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:41.298059    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:41.311572 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:41.311541    4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:41.433840 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:41.433808    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:38:41.748991 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:38:41.748943    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:38:45.780153 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:45.780099    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:50.776513 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:50.776465    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:55.762212 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:55.762165    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:57.602699 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:38:57.602639    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:00.789084 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:00.789046    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:05.790534 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:05.790493    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:10.751030 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:10.750929    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:11.749725 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:39:11.749681    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:39:15.671096 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:15.671040    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:20.669929 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:20.669868    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:25.595775 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:25.595736    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:26.545690 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:26.545656    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:27.007178 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:27.007138    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:30.588470 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:30.588435    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:31.492727 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:31.492689    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"]
Nov 03 00:39:31.597005 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:31.596961    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:33.178623 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:33.178586    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:34.074542 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:34.074504    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88"]
Nov 03 00:39:34.174539 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:34.174506    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb"]
Nov 03 00:39:34.434432 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:34.434339    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:34.597621 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:34.597571    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:35.984047 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:35.983951    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:38.811955 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:38.811911    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:39.074933 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:39.074837    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88" podUID="cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6" containerName="payload-processing" containerID="cri-o://01923b28ea5359cc0f43f2adf86bc477315f567bb36ca24b8a31995cda58641e" gracePeriod=25
Nov 03 00:39:39.175002 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:39.174911    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb" podUID="dc3bd798-95c8-46bc-8189-0a65252d2911" containerName="payload-pre-processing" containerID="cri-o://dbfd29df2f10dc1b1f03d24e0e9d7266d475b959933099cb59d5f2727cfe1f2a" gracePeriod=25
Nov 03 00:39:40.472426 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:40.472393    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:41.748827 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:39:41.748784    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:39:43.256832 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:43.256789    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb" podUID="dc3bd798-95c8-46bc-8189-0a65252d2911" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:39:43.257274 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:43.256789    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88" podUID="cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:39:44.513075 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:44.513037    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:44.677671 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:44.677632    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:46.513068 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.513004    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd" podUID="4a1aea84-5478-487a-9635-fab8b9a81696" containerName="main" containerID="cri-o://9db8df7ef30546701653123329a760867f261033a8e57758a7416e6b8f7941e1" gracePeriod=45
Nov 03 00:39:46.743609 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.743566    4294 generic.go:356] "Generic (PLEG): container finished" podID="4a1aea84-5478-487a-9635-fab8b9a81696" containerID="9db8df7ef30546701653123329a760867f261033a8e57758a7416e6b8f7941e1" exitCode=0
Nov 03 00:39:46.743785 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.743630    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd" event={"ID":"4a1aea84-5478-487a-9635-fab8b9a81696","Type":"ContainerDied","Data":"9db8df7ef30546701653123329a760867f261033a8e57758a7416e6b8f7941e1"}
Nov 03 00:39:46.768536 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.768471    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:39:46.866677 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.866646    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-home\" (UniqueName: \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-home\") pod \"4a1aea84-5478-487a-9635-fab8b9a81696\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") "
Nov 03 00:39:46.866878 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.866693    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-dshm\") pod \"4a1aea84-5478-487a-9635-fab8b9a81696\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") "
Nov 03 00:39:46.866878 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.866725    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a1aea84-5478-487a-9635-fab8b9a81696-kube-api-access-4tcl7\" (UniqueName: \"kubernetes.io/projected/4a1aea84-5478-487a-9635-fab8b9a81696-kube-api-access-4tcl7\") pod \"4a1aea84-5478-487a-9635-fab8b9a81696\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") "
Nov 03 00:39:46.866878 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.866791    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a1aea84-5478-487a-9635-fab8b9a81696-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a1aea84-5478-487a-9635-fab8b9a81696-tls-certs\") pod \"4a1aea84-5478-487a-9635-fab8b9a81696\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") "
Nov 03 00:39:46.866878 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.866817    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-tmp-dir\") pod \"4a1aea84-5478-487a-9635-fab8b9a81696\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") "
Nov 03 00:39:46.866878 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.866868    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-model-cache\") pod \"4a1aea84-5478-487a-9635-fab8b9a81696\" (UID: \"4a1aea84-5478-487a-9635-fab8b9a81696\") "
Nov 03 00:39:46.867210 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.866944    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-home" pod "4a1aea84-5478-487a-9635-fab8b9a81696" (UID: "4a1aea84-5478-487a-9635-fab8b9a81696"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:39:46.867210 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.867132    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-tmp-dir" pod "4a1aea84-5478-487a-9635-fab8b9a81696" (UID: "4a1aea84-5478-487a-9635-fab8b9a81696"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:39:46.867293 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.867237    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-model-cache" pod "4a1aea84-5478-487a-9635-fab8b9a81696" (UID: "4a1aea84-5478-487a-9635-fab8b9a81696"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:39:46.867356 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.867342    4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-home\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:39:46.867410 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.867358    4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-tmp-dir\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:39:46.867410 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.867374    4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-model-cache\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:39:46.869450 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.869427    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-dshm" pod "4a1aea84-5478-487a-9635-fab8b9a81696" (UID: "4a1aea84-5478-487a-9635-fab8b9a81696"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:39:46.869566 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.869546    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a1aea84-5478-487a-9635-fab8b9a81696-tls-certs" pod "4a1aea84-5478-487a-9635-fab8b9a81696" (UID: "4a1aea84-5478-487a-9635-fab8b9a81696"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:39:46.869629 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.869580    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1aea84-5478-487a-9635-fab8b9a81696-kube-api-access-4tcl7" pod "4a1aea84-5478-487a-9635-fab8b9a81696" (UID: "4a1aea84-5478-487a-9635-fab8b9a81696"). InnerVolumeSpecName "kube-api-access-4tcl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:39:46.968284 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.968255    4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a1aea84-5478-487a-9635-fab8b9a81696-dshm\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:39:46.968284 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.968283    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tcl7\" (UniqueName: \"kubernetes.io/projected/4a1aea84-5478-487a-9635-fab8b9a81696-kube-api-access-4tcl7\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:39:46.968284 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:46.968293    4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a1aea84-5478-487a-9635-fab8b9a81696-tls-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:39:47.750536 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:47.750505    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"
Nov 03 00:39:47.751043 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:47.750504    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd" event={"ID":"4a1aea84-5478-487a-9635-fab8b9a81696","Type":"ContainerDied","Data":"7ccf98529571e7c33d56f9e4378598bd4fec0e3bff31a96f18a387ca5677fcaf"}
Nov 03 00:39:47.751043 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:47.750630    4294 scope.go:122] "RemoveContainer" containerID="9db8df7ef30546701653123329a760867f261033a8e57758a7416e6b8f7941e1"
Nov 03 00:39:47.908700 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:47.908661    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"]
Nov 03 00:39:47.956220 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:47.956183    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-aa4c43/simulator-single-main-aa4c43-kserve-58ffc86b56-vc5zd"]
Nov 03 00:39:48.546139 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:48.546110    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a1aea84-5478-487a-9635-fab8b9a81696" path="/var/lib/kubelet/pods/4a1aea84-5478-487a-9635-fab8b9a81696/volumes"
Nov 03 00:39:53.256028 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:53.255962    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb" podUID="dc3bd798-95c8-46bc-8189-0a65252d2911" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:39:53.256569 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:53.256033    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88" podUID="cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:39:57.921256 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:57.921219    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:59.336726 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.336664    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88"
Nov 03 00:39:59.393928 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.393897    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6-kube-api-access-w7nqc\" (UniqueName: \"kubernetes.io/projected/cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6-kube-api-access-w7nqc\") pod \"cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6\" (UID: \"cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6\") "
Nov 03 00:39:59.394127 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.394081    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6-plugins-config-volume\") pod \"cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6\" (UID: \"cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6\") "
Nov 03 00:39:59.394489 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.394462    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6-plugins-config-volume" pod "cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6" (UID: "cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:39:59.396462 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.396432    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6-kube-api-access-w7nqc" pod "cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6" (UID: "cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6"). InnerVolumeSpecName "kube-api-access-w7nqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:39:59.425333 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.425308    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb"
Nov 03 00:39:59.495112 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.495073    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc3bd798-95c8-46bc-8189-0a65252d2911-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3bd798-95c8-46bc-8189-0a65252d2911-plugins-config-volume\") pod \"dc3bd798-95c8-46bc-8189-0a65252d2911\" (UID: \"dc3bd798-95c8-46bc-8189-0a65252d2911\") "
Nov 03 00:39:59.495322 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.495133    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc3bd798-95c8-46bc-8189-0a65252d2911-kube-api-access-blw26\" (UniqueName: \"kubernetes.io/projected/dc3bd798-95c8-46bc-8189-0a65252d2911-kube-api-access-blw26\") pod \"dc3bd798-95c8-46bc-8189-0a65252d2911\" (UID: \"dc3bd798-95c8-46bc-8189-0a65252d2911\") "
Nov 03 00:39:59.495491 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.495464    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:39:59.495491 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.495489    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7nqc\" (UniqueName: \"kubernetes.io/projected/cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6-kube-api-access-w7nqc\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:39:59.495662 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.495525    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3bd798-95c8-46bc-8189-0a65252d2911-plugins-config-volume" pod "dc3bd798-95c8-46bc-8189-0a65252d2911" (UID: "dc3bd798-95c8-46bc-8189-0a65252d2911"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:39:59.497531 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.497511    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3bd798-95c8-46bc-8189-0a65252d2911-kube-api-access-blw26" pod "dc3bd798-95c8-46bc-8189-0a65252d2911" (UID: "dc3bd798-95c8-46bc-8189-0a65252d2911"). InnerVolumeSpecName "kube-api-access-blw26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:39:59.596641 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.596559    4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc3bd798-95c8-46bc-8189-0a65252d2911-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:39:59.596641 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.596590    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blw26\" (UniqueName: \"kubernetes.io/projected/dc3bd798-95c8-46bc-8189-0a65252d2911-kube-api-access-blw26\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:39:59.809818 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.809767    4294 generic.go:356] "Generic (PLEG): container finished" podID="cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6" containerID="01923b28ea5359cc0f43f2adf86bc477315f567bb36ca24b8a31995cda58641e" exitCode=0
Nov 03 00:39:59.809818 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.809811    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88"
Nov 03 00:39:59.810140 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.809827    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88" event={"ID":"cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6","Type":"ContainerDied","Data":"01923b28ea5359cc0f43f2adf86bc477315f567bb36ca24b8a31995cda58641e"}
Nov 03 00:39:59.810140 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.809889    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88" event={"ID":"cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6","Type":"ContainerDied","Data":"611a236beeb8ff778901093e849b3d35e0a183131792dc22f3ed3fae2a0b41aa"}
Nov 03 00:39:59.810140 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.809923    4294 scope.go:122] "RemoveContainer" containerID="01923b28ea5359cc0f43f2adf86bc477315f567bb36ca24b8a31995cda58641e"
Nov 03 00:39:59.811709 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.811682    4294 generic.go:356] "Generic (PLEG): container finished" podID="dc3bd798-95c8-46bc-8189-0a65252d2911" containerID="dbfd29df2f10dc1b1f03d24e0e9d7266d475b959933099cb59d5f2727cfe1f2a" exitCode=0
Nov 03 00:39:59.811844 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.811720    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb"
Nov 03 00:39:59.811844 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.811732    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb" event={"ID":"dc3bd798-95c8-46bc-8189-0a65252d2911","Type":"ContainerDied","Data":"dbfd29df2f10dc1b1f03d24e0e9d7266d475b959933099cb59d5f2727cfe1f2a"}
Nov 03 00:39:59.811844 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.811782    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb" event={"ID":"dc3bd798-95c8-46bc-8189-0a65252d2911","Type":"ContainerDied","Data":"12fa6d8268fbc1f850cb9233c845648032544d4116ddcc3ef06be3fa8536fb5d"}
Nov 03 00:39:59.822512 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.822482    4294 scope.go:122] "RemoveContainer" containerID="01923b28ea5359cc0f43f2adf86bc477315f567bb36ca24b8a31995cda58641e"
Nov 03 00:39:59.822898 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:39:59.822865    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01923b28ea5359cc0f43f2adf86bc477315f567bb36ca24b8a31995cda58641e\": container with ID starting with 01923b28ea5359cc0f43f2adf86bc477315f567bb36ca24b8a31995cda58641e not found: ID does not exist" containerID="01923b28ea5359cc0f43f2adf86bc477315f567bb36ca24b8a31995cda58641e"
Nov 03 00:39:59.823295 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.822910    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01923b28ea5359cc0f43f2adf86bc477315f567bb36ca24b8a31995cda58641e"} err="failed to get container status \"01923b28ea5359cc0f43f2adf86bc477315f567bb36ca24b8a31995cda58641e\": rpc error: code = NotFound desc = could not find container \"01923b28ea5359cc0f43f2adf86bc477315f567bb36ca24b8a31995cda58641e\": container with ID starting with 01923b28ea5359cc0f43f2adf86bc477315f567bb36ca24b8a31995cda58641e not found: ID does not exist"
Nov 03 00:39:59.823295 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.822936    4294 scope.go:122] "RemoveContainer" containerID="dbfd29df2f10dc1b1f03d24e0e9d7266d475b959933099cb59d5f2727cfe1f2a"
Nov 03 00:39:59.832099 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.832078    4294 scope.go:122] "RemoveContainer" containerID="dbfd29df2f10dc1b1f03d24e0e9d7266d475b959933099cb59d5f2727cfe1f2a"
Nov 03 00:39:59.832368 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:39:59.832349    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbfd29df2f10dc1b1f03d24e0e9d7266d475b959933099cb59d5f2727cfe1f2a\": container with ID starting with dbfd29df2f10dc1b1f03d24e0e9d7266d475b959933099cb59d5f2727cfe1f2a not found: ID does not exist" containerID="dbfd29df2f10dc1b1f03d24e0e9d7266d475b959933099cb59d5f2727cfe1f2a"
Nov 03 00:39:59.832427 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.832375    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbfd29df2f10dc1b1f03d24e0e9d7266d475b959933099cb59d5f2727cfe1f2a"} err="failed to get container status \"dbfd29df2f10dc1b1f03d24e0e9d7266d475b959933099cb59d5f2727cfe1f2a\": rpc error: code = NotFound desc = could not find container \"dbfd29df2f10dc1b1f03d24e0e9d7266d475b959933099cb59d5f2727cfe1f2a\": container with ID starting with dbfd29df2f10dc1b1f03d24e0e9d7266d475b959933099cb59d5f2727cfe1f2a not found: ID does not exist"
Nov 03 00:39:59.967808 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:39:59.967715    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88"]
Nov 03 00:40:00.015308 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:00.015270    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-aa4c43-6b8bdc995xkc88"]
Nov 03 00:40:00.077455 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:00.077418    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb"]
Nov 03 00:40:00.081964 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:00.081932    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-aa4c43-c4ccdcnzpb"]
Nov 03 00:40:00.547176 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:00.547150    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6" path="/var/lib/kubelet/pods/cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6/volumes"
Nov 03 00:40:00.547580 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:00.547539    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc3bd798-95c8-46bc-8189-0a65252d2911" path="/var/lib/kubelet/pods/dc3bd798-95c8-46bc-8189-0a65252d2911/volumes"
Nov 03 00:40:08.478411 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:08.478372    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:08.836649 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:08.836618    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:11.748504 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:40:11.748466    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:40:12.060455 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:12.060373    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:12.610183 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:12.610140    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"]
Nov 03 00:40:12.610427 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:12.610403    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx" podUID="2aebb0a8-32c9-409a-beaa-721ce423294b" containerName="istio-proxy" containerID="cri-o://6b7c0e159140021b767563b60f7c727c424577313d7745d1fcecf118d8c44c99" gracePeriod=30
Nov 03 00:40:12.656499 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:12.656447    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:14.485552 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:14.485514    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:14.691487 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:14.691450    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:17.863973 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.863941    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:40:17.878502 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.878448    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-credential-socket\") pod \"2aebb0a8-32c9-409a-beaa-721ce423294b\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") "
Nov 03 00:40:17.878502 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.878484    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-data\") pod \"2aebb0a8-32c9-409a-beaa-721ce423294b\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") "
Nov 03 00:40:17.878667 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.878505    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-token\" (UniqueName: \"kubernetes.io/projected/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-token\") pod \"2aebb0a8-32c9-409a-beaa-721ce423294b\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") "
Nov 03 00:40:17.878667 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.878570    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-envoy\") pod \"2aebb0a8-32c9-409a-beaa-721ce423294b\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") "
Nov 03 00:40:17.878667 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.878616    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-workload-socket\") pod \"2aebb0a8-32c9-409a-beaa-721ce423294b\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") "
Nov 03 00:40:17.878667 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.878651    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-podinfo\") pod \"2aebb0a8-32c9-409a-beaa-721ce423294b\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") "
Nov 03 00:40:17.878891 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.878696    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aebb0a8-32c9-409a-beaa-721ce423294b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aebb0a8-32c9-409a-beaa-721ce423294b-istiod-ca-cert\") pod \"2aebb0a8-32c9-409a-beaa-721ce423294b\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") "
Nov 03 00:40:17.878891 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.878709    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-credential-socket" pod "2aebb0a8-32c9-409a-beaa-721ce423294b" (UID: "2aebb0a8-32c9-409a-beaa-721ce423294b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:17.878891 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.878721    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-data" pod "2aebb0a8-32c9-409a-beaa-721ce423294b" (UID: "2aebb0a8-32c9-409a-beaa-721ce423294b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:17.878891 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.878738    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aebb0a8-32c9-409a-beaa-721ce423294b-kube-api-access-x6s4n\" (UniqueName: \"kubernetes.io/projected/2aebb0a8-32c9-409a-beaa-721ce423294b-kube-api-access-x6s4n\") pod \"2aebb0a8-32c9-409a-beaa-721ce423294b\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") "
Nov 03 00:40:17.878891 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.878767    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-workload-certs\") pod \"2aebb0a8-32c9-409a-beaa-721ce423294b\" (UID: \"2aebb0a8-32c9-409a-beaa-721ce423294b\") "
Nov 03 00:40:17.879187 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.878938    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-workload-socket" pod "2aebb0a8-32c9-409a-beaa-721ce423294b" (UID: "2aebb0a8-32c9-409a-beaa-721ce423294b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:17.879187 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.879062    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aebb0a8-32c9-409a-beaa-721ce423294b-istiod-ca-cert" pod "2aebb0a8-32c9-409a-beaa-721ce423294b" (UID: "2aebb0a8-32c9-409a-beaa-721ce423294b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:40:17.879269 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.879198    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-workload-certs" pod "2aebb0a8-32c9-409a-beaa-721ce423294b" (UID: "2aebb0a8-32c9-409a-beaa-721ce423294b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:17.879269 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.879225    4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-credential-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:40:17.879269 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.879251    4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-data\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:40:17.879269 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.879266    4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-workload-socket\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:40:17.879442 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.879279    4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2aebb0a8-32c9-409a-beaa-721ce423294b-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:40:17.880911 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.880884    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-podinfo" pod "2aebb0a8-32c9-409a-beaa-721ce423294b" (UID: "2aebb0a8-32c9-409a-beaa-721ce423294b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:40:17.880911 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.880891    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-token" pod "2aebb0a8-32c9-409a-beaa-721ce423294b" (UID: "2aebb0a8-32c9-409a-beaa-721ce423294b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:17.881220 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.881196    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aebb0a8-32c9-409a-beaa-721ce423294b-kube-api-access-x6s4n" pod "2aebb0a8-32c9-409a-beaa-721ce423294b" (UID: "2aebb0a8-32c9-409a-beaa-721ce423294b"). InnerVolumeSpecName "kube-api-access-x6s4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:17.881220 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.881206    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-envoy" pod "2aebb0a8-32c9-409a-beaa-721ce423294b" (UID: "2aebb0a8-32c9-409a-beaa-721ce423294b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:17.903876 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.903849    4294 generic.go:356] "Generic (PLEG): container finished" podID="2aebb0a8-32c9-409a-beaa-721ce423294b" containerID="6b7c0e159140021b767563b60f7c727c424577313d7745d1fcecf118d8c44c99" exitCode=0
Nov 03 00:40:17.904030 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.903900    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"
Nov 03 00:40:17.904030 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.903913    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx" event={"ID":"2aebb0a8-32c9-409a-beaa-721ce423294b","Type":"ContainerDied","Data":"6b7c0e159140021b767563b60f7c727c424577313d7745d1fcecf118d8c44c99"}
Nov 03 00:40:17.904030 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.903957    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx" event={"ID":"2aebb0a8-32c9-409a-beaa-721ce423294b","Type":"ContainerDied","Data":"0f9ba59d961b09e0bd92944aab25589dafa9a300c1eb88c03bceadb3f1ed0065"}
Nov 03 00:40:17.904030 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.903993    4294 scope.go:122] "RemoveContainer" containerID="6b7c0e159140021b767563b60f7c727c424577313d7745d1fcecf118d8c44c99"
Nov 03 00:40:17.913861 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.913844    4294 scope.go:122] "RemoveContainer" containerID="2742dffae6baa24f5b7efab77dd1e9856ec2e8eb39a2f20ad960fdcfb6d5bb32"
Nov 03 00:40:17.921844 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.921827    4294 scope.go:122] "RemoveContainer" containerID="6b7c0e159140021b767563b60f7c727c424577313d7745d1fcecf118d8c44c99"
Nov 03 00:40:17.922213 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:40:17.922192    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b7c0e159140021b767563b60f7c727c424577313d7745d1fcecf118d8c44c99\": container with ID starting with 6b7c0e159140021b767563b60f7c727c424577313d7745d1fcecf118d8c44c99 not found: ID does not exist" containerID="6b7c0e159140021b767563b60f7c727c424577313d7745d1fcecf118d8c44c99"
Nov 03 00:40:17.922298 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.922227    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b7c0e159140021b767563b60f7c727c424577313d7745d1fcecf118d8c44c99"} err="failed to get container status \"6b7c0e159140021b767563b60f7c727c424577313d7745d1fcecf118d8c44c99\": rpc error: code = NotFound desc = could not find container \"6b7c0e159140021b767563b60f7c727c424577313d7745d1fcecf118d8c44c99\": container with ID starting with 6b7c0e159140021b767563b60f7c727c424577313d7745d1fcecf118d8c44c99 not found: ID does not exist"
Nov 03 00:40:17.922298 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.922251    4294 scope.go:122] "RemoveContainer" containerID="2742dffae6baa24f5b7efab77dd1e9856ec2e8eb39a2f20ad960fdcfb6d5bb32"
Nov 03 00:40:17.922508 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:40:17.922488    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2742dffae6baa24f5b7efab77dd1e9856ec2e8eb39a2f20ad960fdcfb6d5bb32\": container with ID starting with 2742dffae6baa24f5b7efab77dd1e9856ec2e8eb39a2f20ad960fdcfb6d5bb32 not found: ID does not exist" containerID="2742dffae6baa24f5b7efab77dd1e9856ec2e8eb39a2f20ad960fdcfb6d5bb32"
Nov 03 00:40:17.922570 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.922517    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2742dffae6baa24f5b7efab77dd1e9856ec2e8eb39a2f20ad960fdcfb6d5bb32"} err="failed to get container status \"2742dffae6baa24f5b7efab77dd1e9856ec2e8eb39a2f20ad960fdcfb6d5bb32\": rpc error: code = NotFound desc = could not find container \"2742dffae6baa24f5b7efab77dd1e9856ec2e8eb39a2f20ad960fdcfb6d5bb32\": container with ID starting with 2742dffae6baa24f5b7efab77dd1e9856ec2e8eb39a2f20ad960fdcfb6d5bb32 not found: ID does not exist"
Nov 03 00:40:17.979836 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.979799    4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-token\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:40:17.979836 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.979830    4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-envoy\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:40:17.980071 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.979845    4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2aebb0a8-32c9-409a-beaa-721ce423294b-istio-podinfo\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:40:17.980071 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.979860    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6s4n\" (UniqueName: \"kubernetes.io/projected/2aebb0a8-32c9-409a-beaa-721ce423294b-kube-api-access-x6s4n\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:40:17.980071 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:17.979873    4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aebb0a8-32c9-409a-beaa-721ce423294b-workload-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:40:18.135447 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:18.135356    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"]
Nov 03 00:40:18.141561 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:18.141532    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-aa4c43-openshift-default-79546dd66fffbsx"]
Nov 03 00:40:18.546447 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:18.546415    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aebb0a8-32c9-409a-beaa-721ce423294b" path="/var/lib/kubelet/pods/2aebb0a8-32c9-409a-beaa-721ce423294b/volumes"
Nov 03 00:40:34.023167 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:34.023126    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:34.187473 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:34.187434    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:36.811321 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:36.811280    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:41.750515 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:40:41.750471    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:40:58.230421 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:58.230376    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:58.403132 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:40:58.403095    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:11.749584 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:41:11.749545    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:41:25.066230 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:41:25.066148    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:41.749803 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:41:41.749766    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:41:58.583914 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:41:58.583873    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:11.752963 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:42:11.752924    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:42:19.674373 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:19.674334    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:37.748743 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.748704    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp"]
Nov 03 00:42:37.749287 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.749165    4294 cpu_manager.go:414] "containerMap: removing container" podUID="cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6" containerName="payload-processing"
Nov 03 00:42:37.749287 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.749181    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6" containerName="payload-processing"
Nov 03 00:42:37.749287 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.749211    4294 cpu_manager.go:414] "containerMap: removing container" podUID="2aebb0a8-32c9-409a-beaa-721ce423294b" containerName="istio-proxy"
Nov 03 00:42:37.749287 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.749216    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aebb0a8-32c9-409a-beaa-721ce423294b" containerName="istio-proxy"
Nov 03 00:42:37.749287 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.749225    4294 cpu_manager.go:414] "containerMap: removing container" podUID="2aebb0a8-32c9-409a-beaa-721ce423294b" containerName="istio-proxy"
Nov 03 00:42:37.749287 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.749230    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aebb0a8-32c9-409a-beaa-721ce423294b" containerName="istio-proxy"
Nov 03 00:42:37.749287 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.749237    4294 cpu_manager.go:414] "containerMap: removing container" podUID="4a1aea84-5478-487a-9635-fab8b9a81696" containerName="main"
Nov 03 00:42:37.749287 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.749244    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a1aea84-5478-487a-9635-fab8b9a81696" containerName="main"
Nov 03 00:42:37.749287 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.749253    4294 cpu_manager.go:414] "containerMap: removing container" podUID="dc3bd798-95c8-46bc-8189-0a65252d2911" containerName="payload-pre-processing"
Nov 03 00:42:37.749287 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.749258    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc3bd798-95c8-46bc-8189-0a65252d2911" containerName="payload-pre-processing"
Nov 03 00:42:37.749620 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.749330    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbf767a8-4020-4f8b-b2c1-4abb5ab90aa6" containerName="payload-processing"
Nov 03 00:42:37.749620 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.749342    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aebb0a8-32c9-409a-beaa-721ce423294b" containerName="istio-proxy"
Nov 03 00:42:37.749620 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.749349    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aebb0a8-32c9-409a-beaa-721ce423294b" containerName="istio-proxy"
Nov 03 00:42:37.749620 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.749355    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a1aea84-5478-487a-9635-fab8b9a81696" containerName="main"
Nov 03 00:42:37.749620 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.749360    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc3bd798-95c8-46bc-8189-0a65252d2911" containerName="payload-pre-processing"
Nov 03 00:42:37.752408 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.752392    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp"
Nov 03 00:42:37.851500 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.851463    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp"]
Nov 03 00:42:37.894349 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.894313    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df25fa68-5269-4951-998b-9e9a2d9aed82-openshift-service-ca\") pod \"maas-api-58b7f98cb5-s6wdp\" (UID: \"df25fa68-5269-4951-998b-9e9a2d9aed82\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp"
Nov 03 00:42:37.894505 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.894446    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df25fa68-5269-4951-998b-9e9a2d9aed82-maas-api-tls\") pod \"maas-api-58b7f98cb5-s6wdp\" (UID: \"df25fa68-5269-4951-998b-9e9a2d9aed82\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp"
Nov 03 00:42:37.894505 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.894490    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df25fa68-5269-4951-998b-9e9a2d9aed82-metrics-certs\") pod \"maas-api-58b7f98cb5-s6wdp\" (UID: \"df25fa68-5269-4951-998b-9e9a2d9aed82\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp"
Nov 03 00:42:37.894606 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.894529    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6vwb\" (UniqueName: \"kubernetes.io/projected/df25fa68-5269-4951-998b-9e9a2d9aed82-kube-api-access-q6vwb\") pod \"maas-api-58b7f98cb5-s6wdp\" (UID: \"df25fa68-5269-4951-998b-9e9a2d9aed82\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp"
Nov 03 00:42:37.995431 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.995395    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6vwb\" (UniqueName: \"kubernetes.io/projected/df25fa68-5269-4951-998b-9e9a2d9aed82-kube-api-access-q6vwb\") pod \"maas-api-58b7f98cb5-s6wdp\" (UID: \"df25fa68-5269-4951-998b-9e9a2d9aed82\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp"
Nov 03 00:42:37.995634 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.995462    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df25fa68-5269-4951-998b-9e9a2d9aed82-openshift-service-ca\") pod \"maas-api-58b7f98cb5-s6wdp\" (UID: \"df25fa68-5269-4951-998b-9e9a2d9aed82\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp"
Nov 03 00:42:37.995634 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.995562    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df25fa68-5269-4951-998b-9e9a2d9aed82-maas-api-tls\") pod \"maas-api-58b7f98cb5-s6wdp\" (UID: \"df25fa68-5269-4951-998b-9e9a2d9aed82\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp"
Nov 03 00:42:37.995634 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.995589    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df25fa68-5269-4951-998b-9e9a2d9aed82-metrics-certs\") pod \"maas-api-58b7f98cb5-s6wdp\" (UID: \"df25fa68-5269-4951-998b-9e9a2d9aed82\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp"
Nov 03 00:42:37.996184 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.996160    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df25fa68-5269-4951-998b-9e9a2d9aed82-openshift-service-ca\") pod \"maas-api-58b7f98cb5-s6wdp\" (UID: \"df25fa68-5269-4951-998b-9e9a2d9aed82\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp"
Nov 03 00:42:37.998144 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.998122    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df25fa68-5269-4951-998b-9e9a2d9aed82-metrics-certs\") pod \"maas-api-58b7f98cb5-s6wdp\" (UID: \"df25fa68-5269-4951-998b-9e9a2d9aed82\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp"
Nov 03 00:42:37.998256 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:37.998175    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df25fa68-5269-4951-998b-9e9a2d9aed82-maas-api-tls\") pod \"maas-api-58b7f98cb5-s6wdp\" (UID: \"df25fa68-5269-4951-998b-9e9a2d9aed82\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp"
Nov 03 00:42:38.051148 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:38.051081    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6vwb\" (UniqueName: \"kubernetes.io/projected/df25fa68-5269-4951-998b-9e9a2d9aed82-kube-api-access-q6vwb\") pod \"maas-api-58b7f98cb5-s6wdp\" (UID: \"df25fa68-5269-4951-998b-9e9a2d9aed82\") " pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp"
Nov 03 00:42:38.064962 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:38.064931    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp"
Nov 03 00:42:38.429922 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:38.429897    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp"]
Nov 03 00:42:38.431479 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:42:38.431438    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf25fa68_5269_4951_998b_9e9a2d9aed82.slice/crio-109b44444c14ef070cbdd31f79edee69f578d1822caf6f6770c8e04949676617 WatchSource:0}: Error finding container 109b44444c14ef070cbdd31f79edee69f578d1822caf6f6770c8e04949676617: Status 404 returned error can't find the container with id 109b44444c14ef070cbdd31f79edee69f578d1822caf6f6770c8e04949676617
Nov 03 00:42:38.432749 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:38.432730    4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:42:38.598647 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:38.598610    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp" event={"ID":"df25fa68-5269-4951-998b-9e9a2d9aed82","Type":"ContainerStarted","Data":"109b44444c14ef070cbdd31f79edee69f578d1822caf6f6770c8e04949676617"}
Nov 03 00:42:39.605314 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:39.605279    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp" event={"ID":"df25fa68-5269-4951-998b-9e9a2d9aed82","Type":"ContainerStarted","Data":"f497a760481c0f290d3e04e913cbe40d4b9c2254a5aa9eb936611d8272cb2d10"}
Nov 03 00:42:39.605755 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:39.605419    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp"
Nov 03 00:42:39.715633 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:39.715527    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp" podStartSLOduration=1.741850063 podStartE2EDuration="2.71551334s" totalImagesPullingTime="973.663277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:39.713758901 +0000 UTC m=+6599.282113697" watchObservedRunningTime="2026-11-03 00:42:39.71551334 +0000 UTC m=+6599.283868157"
Nov 03 00:42:41.750524 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:42:41.750485    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:42:45.615139 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:45.615108    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-58b7f98cb5-s6wdp"
Nov 03 00:42:45.776684 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:45.776650    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"]
Nov 03 00:42:45.776900 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:45.776858    4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw" podUID="7a47f445-7bff-4bed-9a6d-ba59d2e0fe18" containerName="maas-api" containerID="cri-o://1183abf138d4c4d5d3e17570644394779e6d86613c5a40f401fbee5a3a9aa368" gracePeriod=30
Nov 03 00:42:47.016393 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.016364    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"
Nov 03 00:42:47.187556 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.187472    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-maas-api-tls\") pod \"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18\" (UID: \"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18\") "
Nov 03 00:42:47.187556 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.187545    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-metrics-certs\") pod \"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18\" (UID: \"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18\") "
Nov 03 00:42:47.187782 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.187570    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-kube-api-access-mghwg\" (UniqueName: \"kubernetes.io/projected/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-kube-api-access-mghwg\") pod \"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18\" (UID: \"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18\") "
Nov 03 00:42:47.187782 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.187606    4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-openshift-service-ca\") pod \"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18\" (UID: \"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18\") "
Nov 03 00:42:47.188024 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.187972    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-openshift-service-ca" pod "7a47f445-7bff-4bed-9a6d-ba59d2e0fe18" (UID: "7a47f445-7bff-4bed-9a6d-ba59d2e0fe18"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:42:47.189830 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.189806    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-metrics-certs" pod "7a47f445-7bff-4bed-9a6d-ba59d2e0fe18" (UID: "7a47f445-7bff-4bed-9a6d-ba59d2e0fe18"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:42:47.189948 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.189825    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-maas-api-tls" pod "7a47f445-7bff-4bed-9a6d-ba59d2e0fe18" (UID: "7a47f445-7bff-4bed-9a6d-ba59d2e0fe18"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:42:47.189948 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.189876    4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-kube-api-access-mghwg" pod "7a47f445-7bff-4bed-9a6d-ba59d2e0fe18" (UID: "7a47f445-7bff-4bed-9a6d-ba59d2e0fe18"). InnerVolumeSpecName "kube-api-access-mghwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:42:47.288485 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.288448    4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:47.288485 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.288480    4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-maas-api-tls\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:47.288485 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.288489    4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-metrics-certs\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:47.288742 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.288499    4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mghwg\" (UniqueName: \"kubernetes.io/projected/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18-kube-api-access-mghwg\") on node \"ip-10-0-139-234.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 00:42:47.644932 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.644896    4294 generic.go:356] "Generic (PLEG): container finished" podID="7a47f445-7bff-4bed-9a6d-ba59d2e0fe18" containerID="1183abf138d4c4d5d3e17570644394779e6d86613c5a40f401fbee5a3a9aa368" exitCode=0
Nov 03 00:42:47.645139 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.644946    4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"
Nov 03 00:42:47.645139 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.645007    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw" event={"ID":"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18","Type":"ContainerDied","Data":"1183abf138d4c4d5d3e17570644394779e6d86613c5a40f401fbee5a3a9aa368"}
Nov 03 00:42:47.645139 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.645068    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65f5668669-dkvfw" event={"ID":"7a47f445-7bff-4bed-9a6d-ba59d2e0fe18","Type":"ContainerDied","Data":"ebd6df4842dc846e4e083e536e57a4548a8af3b431c0ecea3c06c77b37f1f9c2"}
Nov 03 00:42:47.645139 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.645097    4294 scope.go:122] "RemoveContainer" containerID="1183abf138d4c4d5d3e17570644394779e6d86613c5a40f401fbee5a3a9aa368"
Nov 03 00:42:47.654864 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.654846    4294 scope.go:122] "RemoveContainer" containerID="1183abf138d4c4d5d3e17570644394779e6d86613c5a40f401fbee5a3a9aa368"
Nov 03 00:42:47.655140 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:42:47.655121    4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1183abf138d4c4d5d3e17570644394779e6d86613c5a40f401fbee5a3a9aa368\": container with ID starting with 1183abf138d4c4d5d3e17570644394779e6d86613c5a40f401fbee5a3a9aa368 not found: ID does not exist" containerID="1183abf138d4c4d5d3e17570644394779e6d86613c5a40f401fbee5a3a9aa368"
Nov 03 00:42:47.655191 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.655149    4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1183abf138d4c4d5d3e17570644394779e6d86613c5a40f401fbee5a3a9aa368"} err="failed to get container status \"1183abf138d4c4d5d3e17570644394779e6d86613c5a40f401fbee5a3a9aa368\": rpc error: code = NotFound desc = could not find container \"1183abf138d4c4d5d3e17570644394779e6d86613c5a40f401fbee5a3a9aa368\": container with ID starting with 1183abf138d4c4d5d3e17570644394779e6d86613c5a40f401fbee5a3a9aa368 not found: ID does not exist"
Nov 03 00:42:47.802222 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.802185    4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"]
Nov 03 00:42:47.849107 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:47.849071    4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-65f5668669-dkvfw"]
Nov 03 00:42:48.546604 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:48.546574    4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a47f445-7bff-4bed-9a6d-ba59d2e0fe18" path="/var/lib/kubelet/pods/7a47f445-7bff-4bed-9a6d-ba59d2e0fe18/volumes"
Nov 03 00:42:58.933670 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:42:58.933539    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:11.749871 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:43:11.749831    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:43:41.750508 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:43:41.750462    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:43:43.081868 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:43:43.081833    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:59.271349 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:43:59.271312    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:59.434221 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:43:59.434186    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:11.751141 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:44:11.751096    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:44:41.759130 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:44:41.759091    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:44:59.602330 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:44:59.602290    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:11.749038 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:45:11.748997    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:45:19.759166 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:45:19.759114    4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:45:19.759852 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:45:19.759229    4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-lgl4w" podUID="f60ec698-0068-4422-aff1-c8ecbe35f709" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:45:41.765696 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:45:41.765622    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:46:03.301179 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:46:03.301146    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:11.748611 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:46:11.748570    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:46:41.752455 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:46:41.752417    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:47:00.290069 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:47:00.290031    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:11.748194 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:47:11.748158    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:47:41.749171 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:47:41.749130    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:48:00.668731 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:48:00.668693    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:05.998140 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:48:05.998090    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:11.751996 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:48:11.751942    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:48:41.750405 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:48:41.750364    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:49:00.778904 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:00.778865    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-575d76f44f-dlvmr_0cb7b02e-68fb-4619-a81c-42bcb8113a5c/authorino/0.log"
Nov 03 00:49:01.021480 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:01.021441    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:11.749183 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:49:11.749147    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:49:12.903623 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:12.903589    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-669b4bd578-5cj8x_ac0193db-08d3-4abb-9a82-c2199ba08ccf/manager/0.log"
Nov 03 00:49:15.363289 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:15.363258    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4njxm_5b22dd01-17ba-4504-9208-7e3f35ff268b/manager/0.log"
Nov 03 00:49:16.894955 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:16.894919    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-58b7f98cb5-s6wdp_df25fa68-5269-4951-998b-9e9a2d9aed82/maas-api/0.log"
Nov 03 00:49:22.745855 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:22.745821    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-575d76f44f-dlvmr_0cb7b02e-68fb-4619-a81c-42bcb8113a5c/authorino/0.log"
Nov 03 00:49:26.612950 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:26.612888    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cc54b748b-wp6k5_b0db34b7-c04a-4f18-aa19-725f7870c7ed/kube-auth-proxy/0.log"
Nov 03 00:49:28.023234 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:28.023173    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-7s45f_4c9b2053-3218-46fb-bfd7-b193dccc8109/istio-proxy/0.log"
Nov 03 00:49:28.848572 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:28.848540    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-lgl4w_f60ec698-0068-4422-aff1-c8ecbe35f709/istio-proxy/0.log"
Nov 03 00:49:29.079918 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:29.079875    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-r6p52_096035a0-30e5-4124-ad5b-8bfe177aebf1/istio-proxy/0.log"
Nov 03 00:49:29.797849 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:29.797816    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-8499744ffb-z7wfd_8d5fa5dc-4df9-41a3-95e0-55dd180cb73f/payload-pre-processing/0.log"
Nov 03 00:49:30.015004 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:30.014954    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-b45869bf8-wtj54_e94f9d08-36a2-4efb-b95c-3798eb31d227/payload-processing/0.log"
Nov 03 00:49:31.331505 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:31.331475    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-qmm5k_2f2848e7-5412-41cf-a47f-4d33a355e417/main/0.log"
Nov 03 00:49:31.594384 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:31.594309    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-twdqr_6a2abc73-1645-4b27-bb93-bbf3091cde02/main/0.log"
Nov 03 00:49:31.821083 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:31.821051    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-vvvwj_20110d77-0721-4910-bcac-42968b4aa72f/main/0.log"
Nov 03 00:49:32.051196 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:32.051115    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-kj87x_500ba602-25ba-42d0-83ca-a9b8f3bae8fa/main/0.log"
Nov 03 00:49:32.503238 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:32.503210    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-9lmpk_b571835c-e34e-4454-8012-31ab1f83b2bc/main/0.log"
Nov 03 00:49:32.943516 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:32.943418    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-6gg4j_f483c007-731c-42e7-a7f2-9b7877fbba41/main/0.log"
Nov 03 00:49:41.748965 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:49:41.748922    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:49:43.703153 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.703120    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jc4lz/must-gather-5rn9f"]
Nov 03 00:49:43.703609 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.703595    4294 cpu_manager.go:414] "containerMap: removing container" podUID="7a47f445-7bff-4bed-9a6d-ba59d2e0fe18" containerName="maas-api"
Nov 03 00:49:43.703655 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.703610    4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a47f445-7bff-4bed-9a6d-ba59d2e0fe18" containerName="maas-api"
Nov 03 00:49:43.703693 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.703676    4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a47f445-7bff-4bed-9a6d-ba59d2e0fe18" containerName="maas-api"
Nov 03 00:49:43.706700 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.706667    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:43.706896 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.706880    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jc4lz/must-gather-5rn9f"
Nov 03 00:49:43.760232 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.760198    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jc4lz/must-gather-5rn9f"]
Nov 03 00:49:43.805784 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.805746    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jc4lz\"/\"openshift-service-ca.crt\""
Nov 03 00:49:43.805784 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.805755    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jc4lz\"/\"kube-root-ca.crt\""
Nov 03 00:49:43.806034 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.805801    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jc4lz\"/\"default-dockercfg-cz7km\""
Nov 03 00:49:43.807804 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.807782    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cgjxh/must-gather-562jm"]
Nov 03 00:49:43.811593 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.811577    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cgjxh/must-gather-562jm"
Nov 03 00:49:43.860416 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.860381    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cgjxh/must-gather-562jm"]
Nov 03 00:49:43.864618 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.864580    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cgjxh\"/\"kube-root-ca.crt\""
Nov 03 00:49:43.864841 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.864669    4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cgjxh\"/\"openshift-service-ca.crt\""
Nov 03 00:49:43.864841 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.864746    4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cgjxh\"/\"default-dockercfg-hnhxn\""
Nov 03 00:49:43.879421 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.879394    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsrfw\" (UniqueName: \"kubernetes.io/projected/4faec15d-8a9b-4178-8fe0-af5fc5ea7f78-kube-api-access-xsrfw\") pod \"must-gather-5rn9f\" (UID: \"4faec15d-8a9b-4178-8fe0-af5fc5ea7f78\") " pod="openshift-must-gather-jc4lz/must-gather-5rn9f"
Nov 03 00:49:43.879555 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.879458    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4faec15d-8a9b-4178-8fe0-af5fc5ea7f78-must-gather-output\") pod \"must-gather-5rn9f\" (UID: \"4faec15d-8a9b-4178-8fe0-af5fc5ea7f78\") " pod="openshift-must-gather-jc4lz/must-gather-5rn9f"
Nov 03 00:49:43.887052 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.887029    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:43.980520 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.980440    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gdtq\" (UniqueName: \"kubernetes.io/projected/a55f5f9a-7fed-469c-9829-38b7e0955af8-kube-api-access-2gdtq\") pod \"must-gather-562jm\" (UID: \"a55f5f9a-7fed-469c-9829-38b7e0955af8\") " pod="openshift-must-gather-cgjxh/must-gather-562jm"
Nov 03 00:49:43.980520 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.980494    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4faec15d-8a9b-4178-8fe0-af5fc5ea7f78-must-gather-output\") pod \"must-gather-5rn9f\" (UID: \"4faec15d-8a9b-4178-8fe0-af5fc5ea7f78\") " pod="openshift-must-gather-jc4lz/must-gather-5rn9f"
Nov 03 00:49:43.980812 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.980610    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a55f5f9a-7fed-469c-9829-38b7e0955af8-must-gather-output\") pod \"must-gather-562jm\" (UID: \"a55f5f9a-7fed-469c-9829-38b7e0955af8\") " pod="openshift-must-gather-cgjxh/must-gather-562jm"
Nov 03 00:49:43.980812 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.980725    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsrfw\" (UniqueName: \"kubernetes.io/projected/4faec15d-8a9b-4178-8fe0-af5fc5ea7f78-kube-api-access-xsrfw\") pod \"must-gather-5rn9f\" (UID: \"4faec15d-8a9b-4178-8fe0-af5fc5ea7f78\") " pod="openshift-must-gather-jc4lz/must-gather-5rn9f"
Nov 03 00:49:43.980884 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:43.980869    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4faec15d-8a9b-4178-8fe0-af5fc5ea7f78-must-gather-output\") pod \"must-gather-5rn9f\" (UID: \"4faec15d-8a9b-4178-8fe0-af5fc5ea7f78\") " pod="openshift-must-gather-jc4lz/must-gather-5rn9f"
Nov 03 00:49:44.036407 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:44.036376    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsrfw\" (UniqueName: \"kubernetes.io/projected/4faec15d-8a9b-4178-8fe0-af5fc5ea7f78-kube-api-access-xsrfw\") pod \"must-gather-5rn9f\" (UID: \"4faec15d-8a9b-4178-8fe0-af5fc5ea7f78\") " pod="openshift-must-gather-jc4lz/must-gather-5rn9f"
Nov 03 00:49:44.081266 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:44.081228    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a55f5f9a-7fed-469c-9829-38b7e0955af8-must-gather-output\") pod \"must-gather-562jm\" (UID: \"a55f5f9a-7fed-469c-9829-38b7e0955af8\") " pod="openshift-must-gather-cgjxh/must-gather-562jm"
Nov 03 00:49:44.081502 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:44.081351    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gdtq\" (UniqueName: \"kubernetes.io/projected/a55f5f9a-7fed-469c-9829-38b7e0955af8-kube-api-access-2gdtq\") pod \"must-gather-562jm\" (UID: \"a55f5f9a-7fed-469c-9829-38b7e0955af8\") " pod="openshift-must-gather-cgjxh/must-gather-562jm"
Nov 03 00:49:44.081613 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:44.081594    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a55f5f9a-7fed-469c-9829-38b7e0955af8-must-gather-output\") pod \"must-gather-562jm\" (UID: \"a55f5f9a-7fed-469c-9829-38b7e0955af8\") " pod="openshift-must-gather-cgjxh/must-gather-562jm"
Nov 03 00:49:44.137093 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:44.137059    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gdtq\" (UniqueName: \"kubernetes.io/projected/a55f5f9a-7fed-469c-9829-38b7e0955af8-kube-api-access-2gdtq\") pod \"must-gather-562jm\" (UID: \"a55f5f9a-7fed-469c-9829-38b7e0955af8\") " pod="openshift-must-gather-cgjxh/must-gather-562jm"
Nov 03 00:49:44.320740 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:44.320698    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jc4lz/must-gather-5rn9f"
Nov 03 00:49:44.425716 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:44.425682    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cgjxh/must-gather-562jm"
Nov 03 00:49:44.683430 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:44.683401    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jc4lz/must-gather-5rn9f"]
Nov 03 00:49:44.684570 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:49:44.684546    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4faec15d_8a9b_4178_8fe0_af5fc5ea7f78.slice/crio-b2138fee6c41609b31ca34dbdc2fcdf9ea2b20a17d3f625c7168a7e98cb3315c WatchSource:0}: Error finding container b2138fee6c41609b31ca34dbdc2fcdf9ea2b20a17d3f625c7168a7e98cb3315c: Status 404 returned error can't find the container with id b2138fee6c41609b31ca34dbdc2fcdf9ea2b20a17d3f625c7168a7e98cb3315c
Nov 03 00:49:44.686489 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:44.686475    4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:49:44.789016 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:44.788992    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cgjxh/must-gather-562jm"]
Nov 03 00:49:44.791077 ip-10-0-139-234 kubenswrapper[4294]: W1103 00:49:44.791053    4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda55f5f9a_7fed_469c_9829_38b7e0955af8.slice/crio-5cfab186cc7a6fe77734d78e693e2d9ea1dfa34258600c2e9a9e5792ec9bc01c WatchSource:0}: Error finding container 5cfab186cc7a6fe77734d78e693e2d9ea1dfa34258600c2e9a9e5792ec9bc01c: Status 404 returned error can't find the container with id 5cfab186cc7a6fe77734d78e693e2d9ea1dfa34258600c2e9a9e5792ec9bc01c
Nov 03 00:49:45.016913 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:45.016880    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cgjxh/must-gather-562jm" event={"ID":"a55f5f9a-7fed-469c-9829-38b7e0955af8","Type":"ContainerStarted","Data":"5cfab186cc7a6fe77734d78e693e2d9ea1dfa34258600c2e9a9e5792ec9bc01c"}
Nov 03 00:49:45.018085 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:45.018058    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jc4lz/must-gather-5rn9f" event={"ID":"4faec15d-8a9b-4178-8fe0-af5fc5ea7f78","Type":"ContainerStarted","Data":"b2138fee6c41609b31ca34dbdc2fcdf9ea2b20a17d3f625c7168a7e98cb3315c"}
Nov 03 00:49:46.079642 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:46.079603    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:47.033473 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:47.032609    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cgjxh/must-gather-562jm" event={"ID":"a55f5f9a-7fed-469c-9829-38b7e0955af8","Type":"ContainerStarted","Data":"3d1c8945c57312425d2b38fa3a87b1f71a1a76d8097f39170557e02a8d008510"}
Nov 03 00:49:47.033473 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:47.032665    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cgjxh/must-gather-562jm" event={"ID":"a55f5f9a-7fed-469c-9829-38b7e0955af8","Type":"ContainerStarted","Data":"7a2133795c3dc2b6125f289ac1309d42036010ffe9ec220f8736d9fc034d1655"}
Nov 03 00:49:47.146770 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:47.146710    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cgjxh/must-gather-562jm" podStartSLOduration=2.609359976 podStartE2EDuration="4.146693457s" totalImagesPullingTime="1.537333481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:47.14530745 +0000 UTC m=+7026.713662237" watchObservedRunningTime="2026-11-03 00:49:47.146693457 +0000 UTC m=+7026.715048270"
Nov 03 00:49:52.062767 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:52.062729    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jc4lz/must-gather-5rn9f" event={"ID":"4faec15d-8a9b-4178-8fe0-af5fc5ea7f78","Type":"ContainerStarted","Data":"ae4949c8010df22afba3f6444e0de6eb04d69c6271b8a05696a83ce6cf300782"}
Nov 03 00:49:52.063236 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:52.062783    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jc4lz/must-gather-5rn9f" event={"ID":"4faec15d-8a9b-4178-8fe0-af5fc5ea7f78","Type":"ContainerStarted","Data":"53fcf8a9cb09665a6080ad5d7b99368f624c34eabb1ae110700ab9b5b5ed3599"}
Nov 03 00:49:53.923105 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:53.918384    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hlfzd_3a474267-46b1-4759-9018-54960f751878/global-pull-secret-syncer/0.log"
Nov 03 00:49:56.270673 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:56.270639    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:56.602052 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:56.596687    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log"
Nov 03 00:49:56.670000 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:56.669655    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-669b4bd578-5cj8x_ac0193db-08d3-4abb-9a82-c2199ba08ccf/manager/0.log"
Nov 03 00:49:58.615006 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:49:58.613622    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-234.us-west-2.compute.internal_990ba2ba1d2d380f95d805ba47d10a19/haproxy/0.log"
Nov 03 00:50:00.116469 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:00.116436    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4njxm_5b22dd01-17ba-4504-9208-7e3f35ff268b/manager/0.log"
Nov 03 00:50:01.352949 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:01.352915    4294 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6zx4w_a0dfa862-aaf8-4ff1-820e-a610a4d41ba8/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:03.638316 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:03.638270    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-575d76f44f-dlvmr_0cb7b02e-68fb-4619-a81c-42bcb8113a5c/authorino/0.log"
Nov 03 00:50:03.685322 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:03.685282    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-669b4bd578-5cj8x_ac0193db-08d3-4abb-9a82-c2199ba08ccf/manager/0.log"
Nov 03 00:50:05.674266 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:05.674203    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-jc4lz/must-gather-5rn9f" podStartSLOduration=15.687690941 podStartE2EDuration="22.674183196s" totalImagesPullingTime="6.986492255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:52.173567983 +0000 UTC m=+7031.741922772" watchObservedRunningTime="2026-11-03 00:50:05.674183196 +0000 UTC m=+7045.242537984"
Nov 03 00:50:05.675474 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:05.675449    4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"]
Nov 03 00:50:05.680861 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:05.680839    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"
Nov 03 00:50:05.773970 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:05.773927    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"]
Nov 03 00:50:05.790664 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:05.786276    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d409a5a1-09c9-4a6f-8491-da4df51a385b-lib-modules\") pod \"perf-node-gather-daemonset-kwkzp\" (UID: \"d409a5a1-09c9-4a6f-8491-da4df51a385b\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"
Nov 03 00:50:05.790664 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:05.786362    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d409a5a1-09c9-4a6f-8491-da4df51a385b-proc\") pod \"perf-node-gather-daemonset-kwkzp\" (UID: \"d409a5a1-09c9-4a6f-8491-da4df51a385b\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"
Nov 03 00:50:05.790664 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:05.786422    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d409a5a1-09c9-4a6f-8491-da4df51a385b-sys\") pod \"perf-node-gather-daemonset-kwkzp\" (UID: \"d409a5a1-09c9-4a6f-8491-da4df51a385b\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"
Nov 03 00:50:05.790664 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:05.786469    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d409a5a1-09c9-4a6f-8491-da4df51a385b-podres\") pod \"perf-node-gather-daemonset-kwkzp\" (UID: \"d409a5a1-09c9-4a6f-8491-da4df51a385b\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"
Nov 03 00:50:05.790664 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:05.786493    4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxb49\" (UniqueName: \"kubernetes.io/projected/d409a5a1-09c9-4a6f-8491-da4df51a385b-kube-api-access-cxb49\") pod \"perf-node-gather-daemonset-kwkzp\" (UID: \"d409a5a1-09c9-4a6f-8491-da4df51a385b\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"
Nov 03 00:50:05.887223 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:05.887183    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d409a5a1-09c9-4a6f-8491-da4df51a385b-sys\") pod \"perf-node-gather-daemonset-kwkzp\" (UID: \"d409a5a1-09c9-4a6f-8491-da4df51a385b\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"
Nov 03 00:50:05.887450 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:05.887251    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d409a5a1-09c9-4a6f-8491-da4df51a385b-podres\") pod \"perf-node-gather-daemonset-kwkzp\" (UID: \"d409a5a1-09c9-4a6f-8491-da4df51a385b\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"
Nov 03 00:50:05.887450 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:05.887304    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d409a5a1-09c9-4a6f-8491-da4df51a385b-sys\") pod \"perf-node-gather-daemonset-kwkzp\" (UID: \"d409a5a1-09c9-4a6f-8491-da4df51a385b\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"
Nov 03 00:50:05.887450 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:05.887317    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxb49\" (UniqueName: \"kubernetes.io/projected/d409a5a1-09c9-4a6f-8491-da4df51a385b-kube-api-access-cxb49\") pod \"perf-node-gather-daemonset-kwkzp\" (UID: \"d409a5a1-09c9-4a6f-8491-da4df51a385b\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"
Nov 03 00:50:05.887450 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:05.887354    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d409a5a1-09c9-4a6f-8491-da4df51a385b-podres\") pod \"perf-node-gather-daemonset-kwkzp\" (UID: \"d409a5a1-09c9-4a6f-8491-da4df51a385b\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"
Nov 03 00:50:05.887715 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:05.887460    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d409a5a1-09c9-4a6f-8491-da4df51a385b-lib-modules\") pod \"perf-node-gather-daemonset-kwkzp\" (UID: \"d409a5a1-09c9-4a6f-8491-da4df51a385b\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"
Nov 03 00:50:05.887715 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:05.887519    4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d409a5a1-09c9-4a6f-8491-da4df51a385b-proc\") pod \"perf-node-gather-daemonset-kwkzp\" (UID: \"d409a5a1-09c9-4a6f-8491-da4df51a385b\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"
Nov 03 00:50:05.887715 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:05.887572    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d409a5a1-09c9-4a6f-8491-da4df51a385b-lib-modules\") pod \"perf-node-gather-daemonset-kwkzp\" (UID: \"d409a5a1-09c9-4a6f-8491-da4df51a385b\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"
Nov 03 00:50:05.887715 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:05.887612    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d409a5a1-09c9-4a6f-8491-da4df51a385b-proc\") pod \"perf-node-gather-daemonset-kwkzp\" (UID: \"d409a5a1-09c9-4a6f-8491-da4df51a385b\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"
Nov 03 00:50:05.942449 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:05.942374    4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxb49\" (UniqueName: \"kubernetes.io/projected/d409a5a1-09c9-4a6f-8491-da4df51a385b-kube-api-access-cxb49\") pod \"perf-node-gather-daemonset-kwkzp\" (UID: \"d409a5a1-09c9-4a6f-8491-da4df51a385b\") " pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"
Nov 03 00:50:05.998665 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:05.998620    4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"
Nov 03 00:50:06.383971 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:06.383935    4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"]
Nov 03 00:50:06.877340 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:06.877303    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4njxm_5b22dd01-17ba-4504-9208-7e3f35ff268b/manager/0.log"
Nov 03 00:50:07.226220 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:07.226123    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp" event={"ID":"d409a5a1-09c9-4a6f-8491-da4df51a385b","Type":"ContainerStarted","Data":"41f6cd2a7f3c6858d109b82d7828286519d4e94ad95a5a3f63977905e556a610"}
Nov 03 00:50:07.226393 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:07.226360    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"
Nov 03 00:50:07.226440 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:07.226392    4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp" event={"ID":"d409a5a1-09c9-4a6f-8491-da4df51a385b","Type":"ContainerStarted","Data":"645ca44ae4a7107e3209e2193eed7d688f905cedc8789db24c6308c3475a19d3"}
Nov 03 00:50:07.336308 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:07.336236    4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp" podStartSLOduration=2.336216548 podStartE2EDuration="2.336216548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:07.335753785 +0000 UTC m=+7046.904108573" watchObservedRunningTime="2026-11-03 00:50:07.336216548 +0000 UTC m=+7046.904571338"
Nov 03 00:50:09.662349 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:09.662302    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1fc66b40-91ea-4f9f-896e-202f918a6bb1/alertmanager/0.log"
Nov 03 00:50:10.073405 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:10.071604    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1fc66b40-91ea-4f9f-896e-202f918a6bb1/config-reloader/0.log"
Nov 03 00:50:10.184377 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:10.184346    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-669b4bd578-5cj8x_ac0193db-08d3-4abb-9a82-c2199ba08ccf/manager/0.log"
Nov 03 00:50:10.482693 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:10.482599    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1fc66b40-91ea-4f9f-896e-202f918a6bb1/kube-rbac-proxy-web/0.log"
Nov 03 00:50:10.892696 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:10.892660    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1fc66b40-91ea-4f9f-896e-202f918a6bb1/kube-rbac-proxy/0.log"
Nov 03 00:50:11.308870 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:11.308834    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1fc66b40-91ea-4f9f-896e-202f918a6bb1/kube-rbac-proxy-metric/0.log"
Nov 03 00:50:11.717956 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:11.717871    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1fc66b40-91ea-4f9f-896e-202f918a6bb1/prom-label-proxy/0.log"
Nov 03 00:50:11.755178 ip-10-0-139-234 kubenswrapper[4294]: E1103 00:50:11.755142    4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-ngvq8" containerName="extractor"
Nov 03 00:50:12.126218 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:12.126186    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1fc66b40-91ea-4f9f-896e-202f918a6bb1/init-config-reloader/0.log"
Nov 03 00:50:13.247122 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:13.247090    4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cgjxh/perf-node-gather-daemonset-kwkzp"
Nov 03 00:50:13.589149 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:13.589067    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-4njxm_5b22dd01-17ba-4504-9208-7e3f35ff268b/manager/0.log"
Nov 03 00:50:14.199821 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:14.199789    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-74ccc8f57c-bfffb_4f4a11bd-282b-42c6-bb44-943adbbcb4ab/metrics-server/0.log"
Nov 03 00:50:15.023320 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:15.023286    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4smcv_0a8c5208-7b7f-4eb0-a083-bfad09651a37/node-exporter/0.log"
Nov 03 00:50:15.436513 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:15.436407    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4smcv_0a8c5208-7b7f-4eb0-a083-bfad09651a37/kube-rbac-proxy/0.log"
Nov 03 00:50:15.844749 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:15.844724    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4smcv_0a8c5208-7b7f-4eb0-a083-bfad09651a37/init-textfile/0.log"
Nov 03 00:50:19.075120 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:19.075090    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cc54b748b-wp6k5_b0db34b7-c04a-4f18-aa19-725f7870c7ed/kube-auth-proxy/0.log"
Nov 03 00:50:20.140139 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:20.140103    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-7s45f_4c9b2053-3218-46fb-bfd7-b193dccc8109/istio-proxy/0.log"
Nov 03 00:50:20.774717 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:20.774685    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-lgl4w_f60ec698-0068-4422-aff1-c8ecbe35f709/istio-proxy/0.log"
Nov 03 00:50:20.984955 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:20.984922    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-r6p52_096035a0-30e5-4124-ad5b-8bfe177aebf1/istio-proxy/0.log"
Nov 03 00:50:21.616618 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:21.616581    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-8499744ffb-z7wfd_8d5fa5dc-4df9-41a3-95e0-55dd180cb73f/payload-pre-processing/0.log"
Nov 03 00:50:21.826241 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:21.826213    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-b45869bf8-wtj54_e94f9d08-36a2-4efb-b95c-3798eb31d227/payload-processing/0.log"
Nov 03 00:50:23.063012 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:23.062964    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-c4z4j_5fab475f-2b50-4d87-8c50-298ec2a59016/prometheus-operator/0.log"
Nov 03 00:50:23.466470 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:23.466388    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-c4z4j_5fab475f-2b50-4d87-8c50-298ec2a59016/kube-rbac-proxy/0.log"
Nov 03 00:50:24.302603 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:24.302570    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7498985c47-849bp_91387c11-b3d4-46cd-b9a7-fc658f49571e/telemeter-client/0.log"
Nov 03 00:50:24.718161 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:24.718082    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7498985c47-849bp_91387c11-b3d4-46cd-b9a7-fc658f49571e/reload/0.log"
Nov 03 00:50:25.127193 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:25.127162    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7498985c47-849bp_91387c11-b3d4-46cd-b9a7-fc658f49571e/kube-rbac-proxy/0.log"
Nov 03 00:50:27.575620 ip-10-0-139-234 kubenswrapper[4294]: I1103 00:50:27.575587    4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-575d76f44f-dlvmr_0cb7b02e-68fb-4619-a81c-42bcb8113a5c/authorino/0.log"